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FOREWORD 


The  U.S.  Army  Research  Institute  Aviation  Research  and 
Development  Activity  (ARIARDA)  at  Fort  Rucker,  Alabama,  is 
responsible  for  research  and  development  that  will  increase  the 
effectiveness  of  Army  aviator  training.  The  responsibilities 
encompass  training  for  both  Active  Component  (AC)  and  Reserve 
Component  (RC)  aviators . 

As  part  of  the  Army's  "total  force"  concept,  RC  aviators  are 
required  to  train  to  the  same  standards  and  to  maintain  the  same 
levels  of  flight  proficiency  and  flight  safety  as  AC  aviators. 
Since  RC  aviators  must  meet  this  requirement  with  limited  re¬ 
sources,  the  individuals  responsible  for  planning,  implementing, 
and  evaluating  RC  training  must  manage  the  resources  available  to 
them  efficiently.  The  Army  helps  RC  training  managers  achieve 
efficiency  through  evaluation  visits  from  Aviation  Resource 
Management  Survey  (ARMS)  teams. 

This  report  documents  the  results  of  a  questionnaire  survey 
designed  to  evaluate  the  First  Army  ARMS  Checklist  and  the  pro¬ 
cedures  used  to  administer  the  checklist.  The  results  of  the 
survey  provide  information  that  can  be  used  by  the  First  Army 
ARMS  team  to  improve  the  quality  of  the  checklist  and  to  conduct 
evaluation  visits  more  efficiently.  As  part  of  the  research 
effort,  an  information  data  base  was  developed  that  will  help 
Army  managers  organize,  interpret,  and  summarize  the  results  of 
ARMS  visits. 

Mr.  Charles  A.  Gainer,  Chief,  ARIARDA,  Fort  Rucker,  Alabama, 
was  the  technical  monitor *f or  the  project.  The  research  was  com¬ 
pleted  under  the  Letter  of  Agreement  between  the  Deputy  Chief  of 
Staff  for  Training  (DCST) ,  First  U.S.  Army  and  the  Army  Research 
Institute,  Subject;  Aviation  Resource  Management  Survey.  The 
results  of  the  research  were  briefed  to  Colonel  Vay,  First  Army 
DCST,  and  Lieutenant  Colonel  Beasley,  Chief,  Aviation  Division, 
DCST,  on  June  3,  1986,  at  First  Army  Headquarters,  Fort  Meade, 
Maryland,  and  to  staff  members  of  the  Aviation  Division,  DCST,  on 
March  28,  1987,  during  the  First  Army  Aviation  Standardization 
Conference  in  Baltimore,  Maryland.  The  Aviation  Division  per¬ 
sonnel  applied  the  results  of  the  research  when  they  revised  the 
ARMS  Checklist  during  Fiscal  Year  1987. 


V 


This  report  is  divided  into  two  volumes.  Volume  I  contains 
the  primary  report  and  Appendixes  A  and  B.  Volume  II  contains 
Appendix  C  and  the  ARMS  Checklist  Data  Base.  The  data  in  Ap- 

mS  S  are  also  available  on  floppy  disc  in  a  dBASE  III  file  in 

nt>-uob  format. 


EDGAR  M.  JOHNSON 
Technical  Director 
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AN  EVALUATION  OF  THE  AVIATION  RESOURCE  MANAGEMENT  SURVEY  (ARMS) 
CHECKLIST:  VOLUME  I 


EXECUTIVE  SUMMARY 


This  report  describes  the  results  of  research  that  evaluated 
the  First  U.S.  Army  Aviation  Resource  Management  Survey  (ARMS) 
Checklist  and  the  procedures  used  to  administer  the  checklist. 

The  research  was  conducted  by  the  U.S.  Army  Research  Institute 
Aviation  Research  and  Development  Activity  (ARIARDA)  at  the 
request  of  the  First  Army  Deputy  Chief  of  Staff  for  Training 
(DCST) . 


Requirement: 

As  part  of  the  Army's  "total  force"  concept.  Reserve  Com¬ 
ponent  (RC)  aviators  are  required  to  train  to  the  same  standards 
and  to  maintain  the  same  levels  of  flight  proficiency  and  flight 
safety  as  aviators  serving  in  the  Active  Component  (AC) .  Since 
RC  aviators  must  meet  this  requirement  with  limited  resources, 
the  individuals  responsible  for  planning,  implementing,  and 
evaluating  RC  training  must  manage  the  resources  available  to 
them  efficiently. 

The  Army  helps  RC  training  managers  achieve  efficiency 
through  evaluation  visits  made  by  ARMS  teams.  The  general 
purposes  of  the  ARMS,  as  defined  by  the  U.S.  Army  Forces  Command 
(FORSCOM) ,  are  the  following: 

-  to  evaluate  the  management  of  First  Army  National  Guard 
(ARNG)  and  U.S.  Army  Reserve  (USAR)  aviation  programs, 

to  identify  areas  requiring  additional  emphasis,  and 

to  provide  staff  assistance  as  necessary. 

The  First  Army  ARMS  team's  evaluation  efforts  are  guided  by 
a  written  checklist  containing  670  items  organized  into  11  func¬ 
tional  areas  of  evaluation  (e.g.,  safety,  maintenance).  Each 
item  describes  a  deficiency  that  may  result  in  (a)  the  failure  of 
an  aviation  support  facility  to  perform  its  support  mission,  or 
(b)  the  failure  of  an  aviation  unit  to  perform  its  mobilization 
combat  mission.  The  First  Army  DCST  recognized  that  there  may  be 
problems  with  the  current  ARMS  Checklist  and  requested  ARIARDA 's 
assistance  in  evaluating  and  revising  the  checklist. 

This  project  has  three  general  objectives:  (a)  to  perform  a 
systematic  evaluation  of  the  content  of  the  First  U.S.  Army  ARMS 
Checklist,  (b)  to  develop  a  set  of  recommendations  for  improving 
the  ARMS  Checklist  and  the  procedures  used  to  administer  it,  and 
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(c)  to  develop  an  information  data  base  for  organizing  and 
analyzint  ^rms  Checklist  data. 


Procedure : 

A  preliminary  review  of  the  First  Army  ARMS  program  iden¬ 
tified  the  following  specific  problems  in  the  checklist  content 
and  evaluation  procedures: 

-  The  ARMS  Checklist  is  excessively  long.  There  are  many 
items  that  may  not  be  related  to  mission  success. 

-  The  procedures  used  to  evaluate  checklist  items  and  to 
combine  ratings  from  the  various  functional  areas  into  an 
overall  rating  are  not  standardized. 

-  The  negatively  worded  item  format  is  contrary  to  guide¬ 
lines  derived  from  research  on  sentence  comprehension. 

The  items  are  not  listed  in  an  order  that  allows  an 
inexperienced  evaluator  to  proceed  efficiently  through 
the  evaluation  steps. 

-  The  items  are  not  identified  as  specific  to  an  aviation 
facility,  an  aviation  unit,  or  both. 

-  Many  items  are  too  general  to  be  associated  with  observ¬ 
able  conditions  or  events. 

-  There  is  no  systematic  procedure  for  collating  informa¬ 
tion  about  commonly  occurring  deficiencies  observed 
across  facilities  or  units  during  one  year. 

Feedback  from  the  preliminary  review  led  to  the  identifica¬ 
tion  of  three  criteria  that  each  item  should  meet  to  be  on  the 
checklist.  Specifically,  an  item  should  be  retained  only  if  the 
deficiency  addressed  in  the  item  (a)  is  easily  detectable  during 
an  ARMS  visit  (Detectability) ,  (b)  is  important  for  judging  the 
status  of  one  of  the  functional  areas  (Importance) ,  and  (c)  is 
critical  for  mission  success  (Criticality) .  The  extent  to  which 
the  checklist  items  in  each  of  the  functional  areas  meet  the 
three  criteria  was  assessed  by  survey  questionnaires.  A  dif¬ 
ferent  questionnaire  was  developed  for  each  of  the  functional 
areas.  Respondents  to  the  questionnai'^es  were  aviators  and 
aviation  technicians  from  ARNG  and  USAR  aviation  support  facili¬ 
ties  and  aviation  units. 


Findings: 

The  results  indicate  that,  on  the  average,  the  Detectability 
and  Importance  of  the  deficiencies  described  on  the  checklist 
were  rated  moderate  to  high,  while  the  Criticality  was  rated  low. 
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The  low  Criticality  ratings  may  indicate  that  the  majority  of  the 
deficiencies  described  in  the  items  could  exist  in  isolation  in  a 
facility  or  in  a  unit  without  adversely  affecting  the  ability  of 
a  facility  or  a  unit  to  accomplish  its  mission.  It  is  not  pos¬ 
sible  to  conclude  from  the  results  what  the  combined  effect  of 
two  or  more  deficiencies  might  be.  The  rating  distributions  and 
the  rating  scale  verbal  anchors  suggest  that  different  criteria 
may  be  appropriate  for  determining  if  an  item  is  low  or  high  on 
the  Detectability,  Importance,  and  Criticality  scales. 

The  Detectability,  Importance,  and  Criticality  ratings  for  a 
facility  and  for  a  unit  are  very  similar,  suggesting  that  there 
is  no  need  to  develop  different  checklists  for  a  facility  and  a 
unit.  Rather,  a  single  checklist  should  be  developed  in  which 
the  items  that  pertain  only  to  a  facility  or  only  to  a  unit  are 
clearly  identified. 

A  procedure  was  developed  for  using  the  Detectability, 
Importance,  and  Criticality  information  to  decide  whether  to 
retain,  revise,  or  delete  individual  checklist  items.  The 
procedure  should  be  applied  to  the  ratings  for  both  a  facility 
and  a  unit. 

An  ARMS  Checklist  data  base  was  developed  to  summarize  (a) 
the  ARNG  and  USAR  aviators'  ratings  of  the  Detectability,  Impor¬ 
tance,  and  Criticality  of  the  checklist  items,  and  (b)  the 
performance  of  ARNG  and  USAR  units  on  specific  checklist  items 
and  functional  areas  during  future  ARMS  visits.  A  printed  copy 
of  the  data  base  is  included  in  Volume  II  of  this  report;  the 
data  base  is  available  on  floppy  disc  in  a  dBASE  III  file  in 
MS-DOS  format. 


Utilization  of  Findings: 

The  primary  recommendations  of  this  research  are; 

-  The  decision  to  retain,  revise,  or  delete  a  checklist 
item  should  be  based  on  an  assessment  of  the  item's 
Criticality,  Importance,  and  Detectability  ratings  for 
both  a  facility  and  a  unit. 

-  A  single  version  of  the  checklist  should  be  used,  rather 
than  separate  versions  for  a  facility  and  a  unit.  Items 
that  pertain  only  to  a  facility  or  only  to  a  unit  should 
be  identified  on  the  single  checklist. 

-  The  ARMS  Checklist  Data  Base  should  be  used  for  making 
improvements  to  the  checklist  format  and  for  identifying 
commonly  occurring  deficiencies  in  RC  units. 
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GLOSSARY  OF  ACRONYMS  AND  ABBREVIATIONS 


AARM 

AC 

ACR 

AFLO 

ALSE 

AMM 

AR 

ARCOM 

ARIARDA 

ARMS 

ARNG 

ASM 
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CART 

CO 

CONUSA 

DA 

DCST 

DES 

FORSCOM 

FUO 

LOG 

MTFS 

MTM 

MUSARC 

NCO 

NGR 

POC 

POL 

PSEC 

RC 

SAAO 

SIP 

SME 

TRADOC 

USAR 

WO 


Aviation  Armament 
Active  Component 

A'''.ation  Crash,  Rescue,  and  Firefighting 
Aircraft  and  Flightline  Operations 
Aviation  Life  Support  Equipment 
Aeromedical  Management 
Army  Regulation 
Army  Command 

Army  Research  Institute  Aviation  Research  and 
Development  Activity 
Aviation  Resource  Management  Survey 
Army  National  Guard 
Aviation  Safety  Management 
Aviation  Standardization  and  Training 
Centralized  Aviation  Readiness  Team 
Commissioned  Officer 
Continental  U.S.  Army 
Department  of  the  Army 
Deputy  Chief  of  Staff  for  Training 
Directorate  of  Evaluation  and  Standardization 
Forces  Command 
Facility/Unit  Operations 
Aviation  Logistics 

Maintenance  Test  Flight  Standardization 

Maintenance  Management  Training 

Major  U.S.  Army  Reserve  Command 

Noncommissioned  Officer 

National  Guard  Regulation 

Point  of  Contact 

Petroleum,  Oil,  and  Lxibricants 

Physical  Security 

Reserve  Component 

State  Army  Aviation  Officer 

Standardization  Instructor  Pilot 

Subject  Matter  Expert 

Training  and  Doctrine  Command 

United  States  Army  Reserve 

Warrant  Officer 
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AN  EVALUATION  OF  THE  AVIATION  RESOURCE 
MANAGEMENT  SURVEY  (ARMS)  CHECKLIST: 
VOLUME  I 

INTRODUCTION 


According  to  the  Army's  "total  force"  concept,  Reserve 
Component  (RC)  aviators  serving  in  the  U.  S.  Army  Reserve 
(USAR)  and  the  Army  National  Guard  (ARNG)  are  required  to 
train  to  the  same  standards  and  to  maintain  the  same  levels  of 
flight  proficiency  and  flight  safety  as  aviators  serving  in  the 
Active  Component  (AC) .  RC  aviators  must  meet  these  requirements 
with  limited  resources.  Therefore,  the  individuals  who  are 
responsible  for  planning,  implementing,  and  evaluating  RC 
training  must  manage  the  available  resources  (e.g.,  aircraft, 
training  time,  flying  hours,  instructor  pilots)  efficiently. 

One  of  the  ways  that  the  Army  helps  RC  training  managers 
achieve  efficiency  is  through  evaluation  visits  from  Aviation 
Resource  Management  Survey  (ARMS)  teams.  U.  S.  Army  Forces 
Command  (FORSCOM)  Regulation  350-3  (1984)  states  that  the 
general  purposes  of  the  ARMS  are  to  evaluate  the  management  of 
unit  aviation  programs,  to  identify  management  practices  that 
require  improvement,  and  to  provide  staff  assistance  as 
necessary.  As  defined  by  FORSCOM,  the  ARMS  has  four  specific 
objectives: 

-  to  help  commanders  identify  strengths  and  wea)cnesses 
in  all  aviation-related  programs; 

-  to  assess  an  aviation  support  facility's  capacity  to 
support  the  training  of  units  assigned  to  the 
facility; 

-  to  assess  the  aviation  unit's  capabilities  (a)  to 
operate  safely,  efficiently,  and  effectively,  and  (b) 
to  maintain  aviation  resources  apart  from  the 
aviation  support  facility  while  accomplishing  its 
mobilization  mission;  and 

-  to  identify  problems  and  coordinate  assistance 
required  to  solve  problems  that  are  beyond  the 
facility  commander's  or  unit  commander's  sphere  of 
authority . 

The  Deputy  Chief  of  Staff  for  Training  (DCST)  in  each  of 
the  five  Continental  U.  S.  Armies  (CONUSAs)  is  responsible  for 
conducting  ARMS  evaluations.  According  to  FORSCOM  Regulation 
350-3  (1984) ,  an  ARMS  is  to  be  conducted  at  least  once  a  year 
for  each  USAR  facility  and  unit,  and  at  least  once  every  two 
years  for  each  ARNG  facility  and  unit  within  each  CONUSA. 

FORSCOM  formally  established  the  Guide  to  Aviation 
Resources  Management  for  Aircraft  Mishap  Prevention  (1984), 
pxiblished  by  the  U.  S.  Army  Safety  Center,  as  the  standard 
reference  publication  for  the  ARMS.  In  practice,  each  CONUSA 
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uses  its  own  checklist  and  its  own  procedures  for  carrying  out 
its  ASMS  evaluations.  Most  of  the  checklists  are  based  on  the 
Amy  Safety  Center  pviblication.  However,  across  the  CONUSAs, 
there  are  differences  in  the  functional  areas  (e.g.,  safety, 
maintenance)  evaluated,  the  procedures  used  to  assess  the 
status  of  facilities  and  units,  and  the  standards  for 
acceptable  perfomance. 

The  First  Amy  DCST  requested  that  the  U.  S.  Amy 
Research  Institute  Aviation  Research  and  Development  Activity 
(ARIARDA)  provide  assistance  in  evaluating  and  revising  the 
ARMS  Checklist  and  procedures.  The  request  for  assistance  was 
prompted  by  concern  about  problems  with  (a)  the  content  of  the 
checklist,  (b)  the  manner  in  which  the  checklist  items  are  used 
to  evaluate  RC  facilities  and  units,  and  (c)  the  management  and 
utilization  of  infomation  obtained  from  ARMS  visits. 

Preliminary  Review 

ARIARDA  responded  by  conducting  a  preliminary  review  of 
the  First  Amy  ARMS  program.  The  ARIARDA  project  director  met 
with  representatives  of  the  Aviation  Division,  First  Amy  DCST, 
Fort  Meade,  Maryland,  in  June  1985.  The  objectives  of  that 
meeting  were  (a)  to  discuss  the  background  and  purpose  of  the 
ARMS,  (b)  to  review  the  content  of  the  checklist,  and  (c)  to 
discuss  the  procedures  followed  during  an  ARMS  evaluation. 
Subsequently,  in  August  1985,  the  ARIARDA  project  director 
observed  a  First  Amy  ARMS  team  perfoming  an  evaluation  of  the 
USAR  facility  at  Fort  Devens,  Massachusetts.  During  the 
evaluation,  the  techniques  used  to  assess  the  checklist  items 
and  to  detemine  a  rating  of  Satis factory/Unsatisfactory  in 
each  functional  area  were  observed.  In  addition,  the  team 
members  discussed  their  assessments  of  the  checklist  with  the 
project  director.  They  also  provided  suggestions  for 
improving  the  checklist  content  and  its  administrative 
procedures . 

The  discussions  at  Fort  Meade  and  the  observations  at 
Fort  Devens  provided  background  infomation  that  was  essential 
to  the  planning  and  conduct  of  this  research.  A  brief 
description  of  the  First  Amy  Checklist,  the  composition  of  the 
ARMS  team,  and  the  ARMS  evaluation  and  feedback  procedures  are 
presented  below. 

First  Amv  ARMS  Checklist.  The  First  U.  S.  Amy  DCST, 
Aviation  Division,  developed  the  ARMS  Checklist  to  be  used 
during  evaluation  visits.  The  checklist  was  published  in 
October  1983  as  First  Amy  Pamphlet  95-1,  Reserve  Component 
Commander’s  Guide  -  Aviation  Standardization  and  Training 
Program  Evaluation. and  Aviation  Resource  Management  Survey. 
First  Amy  Pamphlet  95-1  subsequently  was  revised  and 
published  again  in  August  1985.  The  checklist  draws  heavily 
from  FORSCOM  FORM  14-1-R,  Reserve  Component  Aviation  Resource 
Management  Survey  Checklist  (1980),  and  the  Amy  Safety  Center 
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publication  referenced  previously.  An  abridged  version  of  the 
First  Army  ARMS  Checklist  is  presented  in  Appendix  A  to 
illustrate  the  content  of  the  checklist.  For  the  sake  of 
brevity,  only  the  introductory  material  and  two  pages  of 
checklist  items  are  included  in  the  abridged  version  to 
illustrate  the  item  format.  A  complete  listing  of  the  checklist 
items  is  included  in  Volume  II  of  this  report. 

The  First  Army  ARMS  Checklist  contains  670  items  that  are 
organized  into  11  functional  areas  of  evaluation.  The 
checklist  items  were  written  by  aviation  subject  matter 
experts  (SMEs)  who  are  knowledgeable  in  each  of  the  functional 
areas  about  (a)  the  operational  requirements  of  RC  support 
facilities,  and  (b)  the  mobilization  mission  requirements  for 
RC  units. 

Table  1  lists  the  functional  areas  in  the  same  order  as 
they  are  presented  in  the  First  Army  ARMS  Checklist.  Table  1 
also  reports  the  number  and  percentage  of  checklist  items 
within  each  functional  area.  Depending  on  the  type  of 
facility  or  unit  visited,  one  or  more  of  the  functional  areas 
may  be  inappropriate  for  evaluation.  For  example,  the  ARMS 

Table  1 

Functional  Areas  And  Items  Contained  in  First  Army  ARMS 
Checklist 


Functional  Area 


Number  Percentage 
of  Items  of  Items 


Aviation  Safety  Management® 
Facility/Unit  Operations® 
Standardization  and  Training 
Aircraft/Flightline  Operations® 
Aeromedical  Management 
Crash,  Rescue,  and  Fire  Fighting 
Petroleum,  Oil,  and  Lubricants 
Maintenance  Management® 

Aviation  Armament 

Aviation  Life  Support  Equipment® 

Physical  Security 


Total 


67 

10.0 

69 

10.3 

99 

14.8 

20 

3.0 

23 

3.4 

37 

5.5 

40 

6.0 

240 

35.8 

24 

3.6 

29 

4.3 

22 

3.3 

670 

100 

®Core  functional  areas. 
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team  would  not  evaluate  Aviation  Armament  (AARM)  during  a  visit 
to  a  facility  that  only  supports  Aeromedlcal  and  Transportation 
units.  Six  of  the  11  areas  (see  Table  1)  are  considered  "core” 
areas  and  are  evaluated  during  every  ARMS  visit. 

Each  checklist  item  describes  a  specific  deficiency  that 
may  result  in  (a)  the  failure  of  a  facility  to  accomplish  its 
mission  of  supporting  its  assigned  RC  units,  or  (b)  the  failure 
of  a  unit  to  accomplish  its  mobilization  combat  mission.  The 
majority  (93%)  of  the  checklist  items  are  worded  as  negative 
statements  (e.g. ,  "The  Aviation  Safety  Officer  was  not  school 
trained.")  rather  than  as  positive  questions  (e.g.,  "Was  the 
Aviation  Safety  Officer  school  trained?")  so  that  they  may  be 
reproduced  verbatim  in  informal  and  formal  reports. 

First  Army  ARMS  team.  The  First  Army  ARMS  team  normally 
consists  of  the  following  core  members:  a  Team  Leader,  a 
Standardization  and  Training  Officer,  an  Aviation  Safety 
Officer,  an  Aviation  Maintenance  Noncommissioned  Officer  (NCO) , 
and  a  Flight  Operations  NCO.  Typically,  each  ARMS  team  member  is 
responsible  for  evaluating  more  than  one  functional  area. 

The  core  members  of  the  evaluation  team  are  supported  by 
Standardization  Instructor  Pilots  (SIPs)  from  the  Directorate  of 
Evaluation  and  Standardization  (DES) ,  U.  S.  Army  Aviation 
Center,  Fort  Rucker,  Alabama.  The  DES  SIPs  evaluate  the 
inflight  performance  of  key  facility  and  unit  aviators  (e.g., 
unit  standardization  pilot,  safety  officer) .  When  required  by 
the  type  of  the  USAR  or  ARNG  facilities  and  units,  the  team  is 
augmented  with  Maintenance  Test  Plight  Evaluators  from  the 
Directorate  of  Evaluation  and  Standardization,  U.  S.  Army 
Aviation  Logistics  School,  Fort  Eustis,  Virginia,  and  by 
technicians  from  the  u.  S.  Army  General  Materiel  and  Petroleum 
Activity,  New  Cumberland  Army  Depot,  Pennsylvania. 

Evaluation  procedures.  During  an  ARMS  visit,  the  ARMS 
team  typically  spends  two  days  evaluating  a  facility  and  two 
days  evaluating  one  of  the  units  training  at  the  facility. 

First,  the  ARMS  team  leader  conducts  an  entrance  briefing  for 
the  facility  or  unit  commander  and  staff  members.  The  ARMS  team 
leader  introduces  the  ARMS  team  members  and  explains  the 
procedures  to  be  followed  during  the  evaluation.  After  the 
entrance  briefing,  the  ARMS  team  members  meet  individually  with 
the  appropriate  facility  or  unit  personnel  to  evaluate  the 
functional  areas,  using  the  ARMS  Checklist  as  a  general  guide. 

At  the  conclusion  of  the  ARMS  evaluation,  a  rating  of 
"Satisfactory"  or  "Unsatisfactory"  is  assigned  to  each  of  the 
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functional  areas  by  the  appropriate  ARMS  team  member.  Using  the 
functional  area  ratings,  the  ARMS  team  leader  assigns  an  overall 
rating  of  "Satisfactory"  or  "Unsatisfactory"  to  a  facility  or 
unit.  According  to  the  First  Army  Pamphlet  95-1,  the  overall 
rating  for  a  facility  or  a  unit  should  consider  the  relative 
significance  of  the  functional  areas  to  (a)  overall  safety 
practices,  (b)  the  degree  to  which  the  facility  or  unit  has 
complied  with  directives,  and  (c)  training  effectiveness  and 
readiness.  Although  there  are  no  strict  guidelines  for  combining 
information  about  the  functional  areas  into  an  overall  rating, 
the  following  two  general  decision  rules  have  been  developed: 

-  A  rating  of  "Unsatisfactory"  on  any  two  of  the  six 
core  areas  identified  in  Table  1  will  result  in  an 
overall  rating  of  "Unsatisfactory." 

-  A  rating  of  "Unsatisfactory"  on  any  one  of  the  core 
areas  and  on  any  two  of  the  remaining  areas  will 
result  in  an  overall  rating  of  "Unsatisfactory." 

Feedback  procedure.  After  an  evaluation  of  a  facility  or 
unit  has  been  completed,  the  members  of  the  ARMS  team  conduct 
an  informal  exit  briefing  and  provide  the  facility  or  unit 
personnel  with  a  copy  of  the  ARMS  Checklist  with  the  observed 
deficiencies  circled.  Upon  concluding  an  ARMS  evaluation 
conducted  for  a  USAR  facility  or  unit  in  a  Major  U.  S.  Army 
Reserve  Command  (MUSARC) ,  the  ARMS  team  conducts  a  formal  exit 
briefing  for  a  designated  representative  of  the  MUSARC 
Commander.  Upon  concluding  an  ARMS  for  an  ARNG  facility  or 
unit  in  a  state,  the  ARMS  team  conducts  a  formal  briefing  for  a 
representative  of  the  state  Adjutant  General.  A  formal  ARMS 
report  is  written  and  sent  to  the  RC  command  personnel  within 
60  days  after  the  evaluation.  The  report  lists  the  specific 
deficiencies  observed  and  recommends  actions  that  should  be 
taken  to  correct  the  deficiencies.  When  appropriate,  the 
report  also  identifies  areas  in  which  the  facility  or  unit 
excelled.  First  Army  requires  that  deficient  facilities  or 
units  submit  Corrective  Action  Plans  indicating  how  specific 
deficiencies  will  be  corrected. 

Copies  of  the  written  ARMS  report  and  the  facility  or 
unit  Corrective  Action  Plan  also  are  sent  to  one  of  the 
Centralized  Aviation  Readiness  Teams  (CART)  within  the  First 
Army  area.  The  mission  of  the  CARTs  is  to  provide  training 
assistance  and  expertise  to  RC  units,  particularly  in  the 
functional  areas  for  which  deficiencies  were  identified  during 
an  ARMS  visit.  Individuals  from  the  ARMS  team  and  the  CARTs 
coordinate  their  activities  to  help  the  RC  facilities  and  units 
to  identify  and  correct  deficiencies.  CART  assistance  is  not 
mandatory,  but  may  be  requested  by  the  individual  RC  facility 
or  unit. 
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Checklist  problems.  During  the  preliminary  review  the 
following  specific  problems  in  the  hecklist  content  and 
evaluation  procedures  were  identified: 

-  The  ASMS  Checklist  is  excessively  long.  There  are 
many  items  that  may  not  be  highly  related  to  mission 
success . 

-  The  procedures  used  to  evaluate  checklist  items  and 
to  combine  ratings  from  the  various  functional  areas 
into  an  overall  rating  are  not  standardized. 

-  The  negatively  worded  item  format  is  contrary  to 
guidelines  derived  from  research  on  sentence 
comprehension.  Carpenter  and  Just  (1975) 
demonstrated  that  sentences  containing  negatives  take 
longer  to  process  than  sentences  containing  only 
positive  assertions.  In  designing  checklists, 
instructions  should  contain  positive  assertions,  if 
possible  (Wickens,  1984) . 

-  The  items  are  not  listed  in  an  order  that  allows  an 
inexperienced  evaluator  to  proceed  efficiently 
through  the  evaluation  steps. 

-  The  items  are  not  identified  as  applicable 
specifically  to  an  aviation  facility,  an  aviation 
unit,  or  both. 

»  Many  items  are  too  general  to  be  associated  with 
observable  conditions  or  events. 

-  There  is  no  systematic  procedure  for  collating 
information  about  commonly  occurring  deficiencies 
observed  across  facilities  or  units  during  one  year. 

Research  Objectives 

To  the  extent  permitted  by  the  available  time  and 
resources,  each  of  the  problems  identified  above  was  addressed 
during  this  research.  The  general  objectives  of  the  ARMS 
Checklist  research  are: 

-  to  perform  a  systematic  evaluation  of  the  content  of 
the  First  U.  S.  Army  ARMS  Checklist, 

-  to  develop  a  set  of  recommendations  for  improving  (a) 
the  ARMS  Checklist  and  (b)  the  procedures  used  to 
administer  it,  and 

-  to  develop  an  information  data  base  for  organizing  and 
analyzing  ARMS  Checklist  data. 
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METHOD 


Overview  of  Research  Approach 

The  results  of  the  preliminary  evaluation  of  the  ARMS 
Checklist  content  and  procedures  were  used  to  formulate  a 
research  approach  for  accomplishing  the  research  objectives. 

The  research  approach  comprised  six  primary  tasks: 

1.  identify  the  checklist  items  to  be  evaluated; 

2.  establish  the  criteria  for  retaining,  revising  or 
deleting  checklist  items; 

3.  obtain  evaluative  judgments  about  the  checklist  items 
from  facility  and  unit  aviation  personnel; 

4.  obtain  evaluative  judgments  about  the  checklist  items 
from  aviation  SMEs; 

5.  recommend  steps  to  improve  the  checklist  content  and 
procedures ;  and 

6.  develop  a  data  base  that  suaunarizes  information  about 
the  checklist  items  and  the  results  from  ARMS 
evaluations. 

As  will  be  described  in  a  later  section,  the  fourth 
research  task  (obtain  aviation  SME  judgments)  was  not 
accomplished  because  of  unavailable  resources.  The  other 
research  tasks  were  accomplished  as  described  in  the  paragraphs 
that  follow. 

Review  and  Revise  Checklist  Items 

During  October  1985,  the  checklist  was  reviewed  to 
identify  items  that  were  no  longer  current  or  relevant  to  any 
of  the  functional  areas.  The  review  was  accomplished  by 
sending  copies  of  the  checklist  to  three  First  Army  ARHG 
facilities  and  to  three  First  Army  USAR  facilities.  A  point  of 
contact  (POC)  was  appointed  by  the  commander  at  each  target 
facility.  Each  POC  instructed  three  or  four  key  staff  members 
(e.g..  Operations  Officer,  Maintenance  Technical  Inspector, 
Safety  Officer)  to  examine  the  checklist  carefully  and  to 
identify  items  that  were  no  longer  current  or  relevant.  The 
facility  POCs  returned  their  copies  of  the  checklist  containing 
the  identified  items  to  the  First  Army  ARMS  team;  the  ARMS  team 
reviewed  the  responses  from  the  ARNG  and  USAR  facilities.  This 
review  resulted  in  the  deletion  of  36  items;  the  remaining  634 
items  were  evaluated  using  the  procedures  described  below. 
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Establish  Checklist  Item  Retention  Criteria 

The  purpose  of  this  task  was  to  establish  the  criteria 
that  each  item  should  meet  to  be  retained  in  the  ARMS 
Checklist.  After  considering  the  intended  purpose  of  the 
checklist,  the  problems  with  the  checklist  described 
previously,  and  the  guidelines  set  forth  in  the  literature  for 
performance  measurement  scales  (e.g.,  Landy  &  Farr,  1983), 
three  criteria  were  established  for  retaining  checklist  items: 

-  The  deficiency  described  in  the  checklist  item  should 
be  detected  without  excessive  effort  during  an  ARMS 
visit  (Detectability) . 

-  The  deficiency  described  in  the  checklist  item  should 
be  weighted  heavily  when  evaluating  the  functional 
area  for  which  it  is  intended,  whether  applied  to  a 
facility,  to  a  unit,  or  to  both  (Importance) . 

The  deficiency  described  in  the  checklist  item  should 
be  deleterious  to  (a)  the  facility's  capability  to 
support  unit  training  or  (b)  the  unit's  capability  to 
perform  its  mobilization  mission  (Criticality) . 

The  researchers  developed  rating  scales  designed  to  collect 
SME  judgments  on  the  Detectability,  Importance,  and  Criticality 
of  each  item.  The  draft  versions  of  the  rating  scales  were 
reviewed  and  critiqued  by  members  of  the  First  Army  ARMS  team 
in  October  1985.  Minor  wording  changes  were  made  to  the  rating 
scales  as  a  result  of  the  review.  The  extent  to  which  each 
item  met  the  three  criteria  was  assessed  by  using  the  rating 
scales  described  in  the  following  paragraphs. 

Detectability.  Detectability  was  defined  as  "the 
relative  ease  or  difficulty  of  determining  during  an  ARMS 
visit  if  the  deficiency  described  in  the  checklist  item  exists 
in  a  facility  or  in  a  unit."  The  respondents  rated  the 
Detectability  of  the  deficiency  described  in  each  checklist 
item  by  responding  to  the  following  rating  question: 

How  much  effort  would  it  take  to  detect  the  deficiency 

described  in  this  item  when  evaluating  a  facility/unit 

during  an  ARMS  visit? 


[1] 

It  would  take 
almost  no  effort 
to  detect  this 
deficiency 


[3] 

It  would  take  a 
moderate  but  not  an 
extensive  amount  of 
effort  to  detect 
this  deficiency 


[4]  [5] 

It  would  take  a 
great  deal  of 
effort  to  detect 
this  deficiency 


[2] 
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The  Detectability  items  were  scaled  on  the  rating  form  such 
that  a  low  score  indicated  a  good  item  and  a  high  score 
indicated  a  poor  item. 


Importance .  Impoz^ance  was  defined  as  "the  amount  of 
weight  that  the  deficiency  described  in  the  item  should  be 
given  when  evaluating  the  status  of  a  facility  or  of  a  unit  in 
a  specific  functional  area."  The  respondents  rated  the 
Importance  of  the  deficiency  described  in  each  checklist  item 
by  responding  to  the  following  rating  question: 


How  much  weight  should  the  deficiency  described  in  this 
item  be  given  when  evaluating  a  functional  area  in  a 
facility/unit? 


[1] 

The  deficiency 
should  be  given 
little  or  no 
weight 


[2]  [3] 

The  deficiency 
should  be  given 
a  moderate  amount 
of  weight 


[4]  [5] 

The  deficiency 
should  be  given 
a  great  deal 
of  weight 


The  Importance  items  were  scaled  on  the  rating  form  such  that  a 
high  score  indicated  a  good  item  and  a  low  score  indicated  a 
poor  item. 


Criticality.  Criticality  was  defined  as  "the  extent  to 
which  a  facility  or  a  unit  with  the  deficiency  would  be 
capable  of  performing  its  mission  in  a  satisfactory  manner." 
The  respondents  rated  the  Criticality  of  the  deficiency 
described  in  each  checklist  item  as  it  applies  to  a  facility 
by  responding  to  the  following  rating  question: 

To  what  extent  could  a  facility  with  the  deficiency 
described  in  this  item  support  the  training  of  a  Reserve 
Component  unit? 


[1]  [2] 
The  facility  could 
support  very  few 
aspects  of  unit 
training 


[3] 

The  facility  could 
support  40-60% 
of  unit 
training 


[4]  [5] 

The  facility  could 
support  nearly  all 
aspects  of  unit 
training 


The  respondents  rated  the  Criticality  of  the  deficiency 
described  in  each  checklist  item  as  it  applies  to  a  unit  by 
responding  to  the  following  rating  question: 


To  what  extent  could  a  unit  with  the  deficiency 
described  in  this  item  perform  its  mobilization  mission 
in  a  satisfactory  manner? 
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[1]  [2]  [3]  [4]  [5] 

The  unit  could  The  unit  could  The  unit  could 

perform  very  few  perform  40-60%  perform  almost  all 
of  its  mobiliza-  of  its  mobiliza-  of  its  mobiliza¬ 
tion  tasks  tion  tasks  tion  tasks 

The  Criticality  items  were  scaled  on  the  rating  form  such  that 
a  low  score  indicated  a  good  item  and  a  high  score  indicated  a 
poor  item.  The  directions  in  which  the  Detectability  and 
Criticality  items  were  scaled  were  different  from  the 
Importance  items  to  minimize  the  effect  of  response  bias. 


The  checklist  items  were  grouped  into  the  appropriate 
functional  areas  and  assembled  into  prototype  rating  booklets. 

To  keep  the  rating  booklets  to  a  manageable  length,  the  items 
in  two  of  the  functional  areas  were  subdivided  into  smaller 
groups  of  items.  Specifically,  the  items  in  the  Standardization 
and  Training  functional  area  were  divided  into  one  group  of 
maintenance  test  flight  standardization  items  and  another  group 
of  standardization  and  training  items.  In  a  similar  manner,  the 
items  in  the  Maintenance  Management  functional  area  were  divided 
into  groups  of  maintenance  management  training  items, 
maintenance  quality  control  items,  maintenance  shop  operations 
items,  and  aviation  logistics  items.  This  item  grouping 
resulted  in  a  new  total  of  fifteen  functional  areas.  A  separate 
booklet,  containing  the  appropriate  checklist  items,  was 
developed  for  each  of  the  fifteen  functional  areas. 

The  first  two  pages  of  each  rating  booklet  contained  the 
rating  instructions  and  Privacy  Act  statement.  On  the  third 
page  of  each  rating  booklet,  the  respondents  were  instructed 
to  provide  the  last  four  digits  of  their  social  security 
number.  The  four-digit  identifier  was  used  for  administrative 
management  of  the  data.  The  respondents  also  were  asked  to 
provide  the  following  military  demographic  information: 

-  category  of  present  duty  position, 

-  years  in  present  duty  position, 

-  CONUSA  to  which  assigned, 

-  total  years  of  military  service, 

-  total  number  of  military  flight  hours, 
functional  area  of  greatest  expertise, 

-  years  of  service  in  the  ARMG,  and 

-  years  of  service  in  the  USAH. 


10 


The  rating  scale  definitions  were  listed  on  the  fourth  page 
of  each  rating  booklet.  The  respondents  were  instructed  to 
read  the  definitions  carefully  before  proceeding  to  the  rating 
task  and  to  review  the  definitions  as  necessary.  In  addition, 
the  respondents  were  instructed  to  assume  that  the  deficiency 
described  in  the  item  was  the  only  deficiency  that  existed  in  a 
facility  or  in  a  unit.  Two  sample  items  were  provided  on  the 
next  two  pages  of  each  booklet. 

Each  of  the  remaining  pages  in  the  booklet  listed  the 
specific  item  to  be  rated,  the  functional  area,  and  the  three 
scales  as  they  apply  first  to  a  facility  and  second  to  a  unit. 

An  alphanumeric  identifier  was  placed  at  the  bottom  right-hand 
portion  of  each  page  for  administrative  management  of  the  forms. 

An  abridged  version  of  the  rating  booklet  for  the  Aircraft 
and  Flightline  Operations  functional  area  is  presented  in 
Appendix  B  to  illustrate  the  content  of  the  rating  booklets. 

For  the  sake  of  brevity,  only  the  introductory  material  and  two 
of  the  Aircraft  and  Flightline  Operations  rating  items  are  shown 
in  the  abridged  version. 

Pretesting  the  Rating  Booklets 

During  weekend  drill  periods  in  November  1985,  the 
prototype  rating  booklets  were  pretested  with  three  aviators 
from  the  345th  Army  Security  Agency  company,  79th  Army  Command 
(ARCOM) ,  Willow  Grove  Naval  Air  Station,  Pennsylvania,  and  with 
three  aviators  from  the  327th  Aviation  Company,  97th  ARCOM, 

Fort  Meade,  Maryland.  The  aviators  were  told  the  purpose  of 
the  research  project  and  were  asked  to  complete  a  prototype 
rating  booklet.  The  aviators  then  were  asked  specific 
questions  about  their  interpretations  of  the  rating  items  and 
were  encouraged  to  suggest  revisions  to  the  content  and  format 
of  the  prototype  rating  booklets.  Members  of  the  research  team 
used  the  information  obtained  during  the  pretest  to  make  minor 
changes  to  the  prototype  rating  booklets.  No  additional 
changes  were  made  to  the  rating  scales.  The  rating  booklets 
then  were  produced  in  final  form. 

Administration  of  the  Rating  Booklets 

The  booklets  containing  the  checklist  rating  items  were 
mailed  to  the  First  Army  ARNG  State  Aviation  Officers  (SAAOs) 
and  to  the  MUSARC  commanders  during  March  1986.  A  sufficient 
number  of  rating  booklets  were  distributed  to  enable  a 
representative  from  each  facility  and  a  I'epresentative  from  one 
of  the  units  assigned  to  each  facility  to  complete  a  booklet 
for  each  functional  area.  The  booklets  were  accompanied  by  a 
cover  letter  from  the  First  Army  DOST  explaining  the  purpose  of 
the  project.  The  SAAOs  and  MUSARC  commanders,  in  turn, 
distributed  the  rating  booklets  to  ARNG  and  USAR  aviators  and 
nonrated  aviation  personnel  (e.g.,  maintenance  technical 
inspectors)  who  were  responsible  for  managing  one  or  more  of 
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the  functional  areas  covered  in  the  AKMS  Checklist.  In  some 
cases,  a  respondent  completed  a  booklet  for  more  than  one  area, 
but  only  if  (a)  the  respondent  possessed  sufficient  expertise 
in  the  areavS)«  and  (b)  another  qualified  respondent  was  not 
available.  The  rating  booklets  were  completed  by  ARNG  and 
USAR  aviation  personnel  during  April  and  May  1986;  they  were 
then  returned  to  ARIARDA  for  processing  and  data  analysis. 

Revision  of  the  Project  Scope 

As  noted  previously,  the  research  approach  required  that 
the  Detectability,  Importance,  and  Criticality  of  the 
checklist  items  be  rated  by  ARNG  and  USAR  aviation  personnel 
and  by  a  group  of  aviation  SMEs.  The  aviation  SMEs  were 
intended  to  be  (a)  members  of  ARMS  teams  and  CARTs  from  the 
six  CONUSAs,  and  (b)  technical  specialists  from  the  DES  at  the 
U.  S.  Army  Aviation  Center,  the  U.  S.  Army  General  Materiel  and 
Petroleum  Activity,  and  the  U.  S.  Army  Safety  Center.  However, 
due  to  the  unavailability  of  the  aviation  SMEs,  ratings  were 
collected  only  from  ARNG  and  USAR  facility  and  unit  aviation 
personnel . 

Development  of  the  ARMS  Checklist  Data  Base 

To  meet  the  third  project  objective,  an  ARMS  Checklist 
data  base  was  developed.  The  data  base  was  designed  (a)  to 
summarize  Information  about  each  of  the  checklist  items,  and 
(b)  to  serve  as  a  tool  for  organizing  the  checklist  items  into 
a  format  that  will  facilitate  the  ARMS  evaluations.  The 
following  information  was  incorporated  into  the  data  base: 

-  a  unique  alphanumeric  item  identifier; 

-  the  DCST  Aviation  Division  word  processing  glossary 
code; 

-  the  name  of  the  checklist  item; 

-  the  functional  area  under  which  the  item  is 
classified; 

-  the  functional  subarea; 

-  the  rating  results  for  facilities  and  units; 

-  the  paragraph (s)  and  the  number (s)  of  the 
publication (s)  used  to  establish  the  evaluative 
standard  for  the  item;  and 

-  the  full  name(s)  of  the  pviblication(s)  referenced. 

A  printed  copy  of  the  data  base  is  presented  as  Appendix  c  in 
Volume  II  of  this  report.  The  data  in  Appendix  C  also  are 
available  on  floppy  disc  in  a  dBASE  III  file  in  MS-DOS  format. 
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A  recoinnended  procedure  for  revising  the  checklist  was 
developed  as  part  of  this  research.  The  procedure  includes  a 
set  of  decision  rules  for  deciding  whether  to  retain,  revise, 
or  delete  checklist  items.  Implementation  of  the  decision 
rules  is  based  on  the  Detectability,  Importance,  and 
Criticality  ratings  of  the  individual  checklist  items.  The 
procedure  for  revising  the  checklist,  including  the  decision 
rules,  is  presented  in  a  separate  section  following  the  Results 
section  of  this  report. 
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RESULTS 


Deinocrraphic  Characteristics  of  the  Respondents 


A  total  of  345  rating  booklets  was  returned  from  259 
aviation  personnel.  As  noted  previously,  some  respondents 
completed  booklets  in  more  than  one  functional  area. 
Approximately  70%  of  the  booklets  were  from  the  ARNG  and  30% 
were  from  the  USAR.  Table  2  presents,  by  functional  area,  the 
percentage  of  respondents  for  five  types  of  duty  positions. 

The  first  column  in  Table  2  shows  the  number  of  respondents  in 
each  functional  area.  The  second  and  third  columns  present  the 


Table  2 

Percentage  of  Respondents  in  Each  Functional  Area  from  ARNG 
and  USAR  Aviation  Facilities  or  Units 


Facility  Unit 
Technicians  Personnel 

Functional  Area  n^  ARNG  USAR  ARNG  USAR  Other 


Aviation  Safety  Management 
Facility/Unit  Operations 
Standardization  and  Training 
Maintenance  Test  Flights 
Aircraft/Flighti ine  Operations 
Aeromedical  Management 
Crash,  Rescue,  and  Firefighting 
Petroleum,  Oil,  and  Lubricants 
Maintenance  Management  Training 
Maintenance  Quality  Control 
Maintenance  Shop  Operations 
Aviation  Logistics 
Aviation  Armament 
Aviation  Life  Support  Equipment 
Physical  Security 


25 

56 

8 

20 

12 

4 

27 

56 

7 

4 

14 

19 

28 

48 

14 

7 

10 

21 

25 

48 

8 

16 

8 

20 

22 

54 

5 

9 

14 

18 

17 

41 

0 

29 

12 

18 

12 

75 

0 

8 

8 

9 

27 

62 

0 

23 

8 

7 

28 

36 

14 

18 

14 

18 

23 

61 

13 

9 

13 

4 

26 

68 

12 

4 

4 

12 

24 

58 

8 

21 

13 

0 

12 

36 

0 

27 

18 

19 

26 

54 

8 

12 

12 

14 

23 

56 

12 

8 

8 

16 

^Nximber  of  respondents  in  each  functional  area. 
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the  percentage  of  respondents  who  were  full-time  ARNG  or  USAR 
aviation  facility  technicians.^  The  fourth  and  fifth  columns 
present  the  percentage  of  respondents  who  were  members  of  ARNG 
or  USAR  units,  but  were  not  full-time  technicians.  Finally,  the 
last  column  in  Table  2  presents  the  percentage  of  respondents 
who  classified  themselves  in  other  types  of  duty  positions.  In 
general,  the  majority  of  respondents  were  ARNG  facility 
technicians;  USAR  facility  technicians  provided  the  fewest 
responses . 

Table  3  shows  the  percentage  of  respondents,  by  functional 
area,  who  were  noncommissioned  officers,  warrant  officers,  or 
commissioned  officers.  The  percentage  of  respondents  in  each 
grade  varied  widely  among  the  functional  areas  of 
responsibility. 

Table  3 


Percentage  of  Noncommissioned  Officer  (NCO) ,  Warrant  Officer 
(WO)  ,  or  Commissioned  Officer  (CO)  ^spondents 


Functional  Area 

NCO 

WO 

CO 

Aviation  Safety  Management 

25 

4 

72 

24 

Facility/Unit  Operations 

27 

7 

26 

67 

Standardization  and  Training 

28 

0 

44 

56 

Maintenance  Test  Flights 

25 

0 

60 

40 

Aircraft/Flightline  Operations 

22 

27 

32 

41 

Aeromedical  Management 

17 

35 

18 

47 

Crash,  Rescue  and  Firefighting 

12 

45 

36 

19 

Petroleum,  Oil,  and  Lubricants 

27 

63 

26 

11 

Maintenance  Management  Training 

28 

24 

64 

12 

Maintenance  Quality  Control 

23 

57 

21 

22 

Maintenance  Shop  Operations 

26 

46 

35 

19 

Aviation  Logistics 

24 

48 

10 

42 

Aviation  Armament 

12 

50 

8 

42 

Aviation  Life  Support  Equipment 

26 

62 

19 

19 

Physical  Security 

23 

4 

61 

35 

^Number  of  respondents  in  each  functional  area. 


^Aviators  who  hold  positions  as  full-time  federal  facility 
technicians  in  the  ARNG  or  the  USAR  are  required  to  belong  to  a 
unit.  Respondents  in  this  category  are  included  in  the 
percentages  of  respondents  reported  in  the  second  and  third 
columns  of  Table  2. 
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The  median  nvimber  of  years  of  military  service  for  all 
respondents  in  various  functional  areas  ranged  from  16.6  years 
(Aeromedical  Management)  to  21.0  years  (Maintenance  Shop 
Operations),  with  an  overall  median  of  18.3  years.  The  median 
number  of  years  that  the  respondents  from  the  various  functional 
areas  had  spent  in  their  present  duty  position  ranged  from  1.7 
years  (Physical  Security)  to  8.0  years  (Maintenance  Quality 
Control),  with  an  overall  median  of  3.4  years. 


This  section  presents  the  results  of  the  Detectability, 
Importance,  and  Criticality  ratings  of  the  checklist  items. 
Following  a  description  of  the  rating  scale  treatment,  the  data 
are  presented  for  the  individual  items  and  in  susunary  form  for 
each  functional  area. 

Rating  scale  treatment.  To  make  the  Interpretation  of 
the  ratings  consistent  for  all  three  scales,  the  original 
Detectability  snd  Criticality  ratings  were  transposed  so  that  a 
low  rating  category  indicates  a  poor  item  (i.e.,  one  that 
probably  should  be  considered  for  revision  or  deletion  from  the 
checklist)  and  a  high  rating  category  indicates  a  good  item 
(i.e.,  one  that  probably  should  be  retained  in  the  checklist). 

The  Importance  ratings  were  already  scaled  in  the  appropriate 
direction.  All  subsequent  data  on  each  scale  are  presented  in 
the  low-to-high,  poor-to-good  format. 

Examination  of  the  rating  scale  distributions  for  the 
individual  checklist  items  (see  Appendix  C,  Volume  II) 
indicates  that  the  ratings  are  not  nonnally  distributed,  thus 
precluding  the  use  of  parametric  statistics  (e.g. ,  the  mean  and 
standard  deviation)  to  describe  the  rating  data.  Instead,  the 
percentage  of  respondents  in  each  rating  scale  category  (1  = 
low;  5  high)  is  presented  for  each  item. 

Item  level  data.  The  percentage  of  responses  in  each 
rating  scale  category  and  the  number  of  respondents  for  each 
item  on  the  Detectability,  Importance,  and  Criticality  scales 
are  presented  in  the  ARMS  Checklist  Data  Base  (see  Appendix  C, 
Volume  II) .  The  ratings  are  presented  separately  for  the  RC 
facilities  and  units.  The  items  in  Appendix  C  are  organized 
into  functional  areas;  the  functional  areas  and  the  items  within 
each  functional  area  are  listed  in  the  same  order  as  in  the  ARMS 
Checklist 

Figure  1  illustrates  the  format  used  for  each  item  in  the 
data  base.  As  discussed  in  the  Method  section  (see  p.  12) ,  the 
items  are  identified  by  both  an  ARIARDA  data  processing 
identifier  (Item)  and  a  First  Army  word  processing  code  (Code)  to 
facilitate  cross  referencing.  The  codes  are  followed  by  the 
functional  subarea,  the  name  of  the  item,  the  facility  and  unit 


Item;  ASMOOl  Code;  226/c  Subarea :  Aviation  Safety  Officer 
Name;  An  Aviation  Safety  Officer  had  not  been  authorized/assigned 


Rating  Category 

1 

Z 

1 

1 

5. 

Facility 

Detectability 

4 

0 

0 

8 

88 

Ratings ; 

Importance 

4 

0 

8 

8 

79 

(n  »  24) 

Criticality 

38 

21 

29 

4 

8 

Unit 

Detectability 

0 

0 

5 

14 

81 

Ratings; 

Importance 

0 

0 

10 

19 

71 

(n  =  21) 

Criticality 

33 

14 

43 

5 

5 

Publication 
Number fs) ; 

Publication 
Name (s) ; 

para  l-6d,  AR  385-95;  para  l-4d,  NCR  385- 
FORSCOM/TRADOC  Suppl  1  to  AR  385-95 

Army  Aviation  Accident  Prevention;  Army 
National  Guard  Safety  Program 

-10; 

Figure  1.  Example  of  item  siunmary  information  from  ARMS 
Checklist  Data  Base. 


ratings  for  each  scale,  and  data  on  the  publications  used  to 
establish  the  evaluative  standards  for  the  item. 

In  item  ASMOOl,  the  results  for  the  facility  and  unit 
ratings  are  very  similar.  The  deficiency  identified  by  the 
item  is  rated  as  very  easy  for  the  ARMS  team  to  detect,  and 
the  item  is  rated  as  very  important  in  the  evaluation  of  the 
functional  area.  The  deficiency  is  rated  as  slightly  to 
moderately  critical  for  mission  performance,  and  slightly  more 
critical  for  a  unit  than  for  a  facility. 

Facility/unit  comparisons.  Figures  2  through  7  graphically 
show  the  response  percentage  distributions  averaged  across 
functional  areas  for  the  Detectability,  Importance,  and 
Criticality  scales.  Figures  2,  4,  and  6  present  the  facility 
distributions;  Figures  3,  5,  and  7  present  the  unit 
distributions . 

Two  conclusions  can  be  drawn  from  the  data  presented  in 
Figures  2  through  7.  First,  within  each  rating  scale,  the 
response  percentage  distributions  for  facilities  are  almost 
identical  to  those  for  the  units.  As  a  result,  the  facility 
and  unit  data  are  combined  in  Tables  4,  5,  and  6.  Second,  the 
distributions  for  the  Detectability  and  Importance  scales  are 
similar  to  each  other  but  differ  markedly  from  the  distributions 
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Figure  2.  Detectability  scale  response  percentage 
distributions  for  the  facility  ratings. 
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Figure  3.  Detectability  scale  response  percentage 
distributions  for  the  unit  ratings. 
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Figure  4 . 


Importance  scale  response  percentage  distributions 
for  the  facility  ratings. 
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Figure  5. 


Importance  scale  response  percentage  distributions 
for  the  unit  ratings. 
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Figure  6. 


Criticality  scale  response  percentage 
distributioris  for  the  facility  ratings. 


Response 

Percentage 


too 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 


The  unit  could  perform 
almost  all  of  its 
mobilization  tasks 


The  unit  could  perform 
40-60%  of  its 
mobilization  tasks 


The  unit  could  perform 
very  few  of  its 
mobilization  tasks 


Figure  7.  Criticality  scale  response  percentage 
distributions  for  the  unit  ratings. 
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for  the  Criticality  scale.  The  responses  on  the  Detectability 
and  Importance  scales  occur  primarily  in  the  middle  (ranging 
from  25%  to  30%)  and  highest  rating  categories  (ranging  from 
33%  to  43%) .  Approximately  7%  of  the  responses  on  the 
Importance  and  Detectability  scales  occur  in  the  lowest  rating 
category.  In  comparison,  approximately  45%  of  the  responses  on 
the  Criticality  scale  occur  in  the  lowest  rating  category; 
approximately  20%  occur  in  categories  2  and  3;  and  approximately 
7%  occur  in  categories  4  and  5.  The  differences  in  response 
percentage  distributions  indicate  that  the  Criticality  scale 
should  be  treated  differently  than  the  Detectability  and 
Importance  scales. 

Functional  area  level  data.  Tables  4  through  6  present 
the  response  percentages  in  each  rating  category,  averaged 
across  items  in  each  functional  area  and  across  facilities  and 
units,  for  the  Detectability,  Importance,  and  Criticality 
scales.  As  depicted  by  the  data  in  Tables  4  through  6, 
substantial  differences  exist  in  the  category  response 
percentages  between  the  individual  functional  areas  for  each  of 
the  three  scales.  However,  the  same  general  pattern  of 
responses  occurs  between  the  individual  functional  areas  for 
each  scale.  That  is,  the  Detectability  and  Importance  scales 
are  negatively  skewed  and  the  Criticality  scale  is  positively 
skewed.  The  differences  in  response  percentages  between  the 
three  scales  (see  the  line  titled  "Across  Functional  Areas"  in 
Tables  4-6)  are  much  greater  than  the  differences  between 
functional  areas  for  each  scale. 


22 


Table  4 


Detectability  Scale  Response  Percentages 


Functional  Area 

Aviation  Safety  Management 

6.0 

8.5 

30.9 

17.6 

37.3 

Pacility/Unit  Operations 

4.6 

6.4 

22.1 

23.8 

43.0 

Standardization  and  Training 

5.6 

6.4 

24.1 

20-.  5 

43.4 

Maintenance  Test  Flights 

4.1 

10.3 

25.1 

16.2 

44.3 

Aircraft/Flight line  Operations 

2.1 

5.6 

25.0 

16.1 

51.3 

Aeromedical  Management 

1.9 

1.3 

35.0 

10.4 

51.3 

Crash,  Rescue,  and  Firefighting 

4.6 

14.9 

37.0 

12.8 

30.6 

Petroleum,  Oil,  and  Lubricants 

2.0 

3.0 

22.1 

17.4 

55.5 

Maintenance  Management  Training 

5.5 

5.6 

31.2 

19.0 

38.6 

Maintenance  Quality  Control 

10.3 

12.7 

35.8 

14.6 

26.7 

Maintenance  Shop  Operations 

3.0 

2.4 

13.9 

14.9 

65.6 

Aviation  Logistics 

11.6 

15.3 

23.8 

18.3 

30.9 

Aviation  Armament 

9.7 

11.6 

24.5 

22.4 

31.1 

Aviation  Life  Support  Equipment 

9.7 

9.6 

19.8 

17.7 

43.1 

Physical  Security 

7.0 

6.1 

17.8 

21.8 

47.2 

Across  Functional  Areas 

5.9 

8.0 

25.9 

17.6 

42.7 

Table  5 

Importance  Scale  Response  Percentages 

HH 

Functional  Area 

1 

2 

3 

4 

5 

Aviation  Safety  Management 

7.3 

12.7 

34.0 

18.1 

28.6 

Facility/Unit  Operations 

6.6 

7.4 

32.8 

24.6 

28.7 

Standardization  and  Training 

8.2 

15.4 

31.3 

22.4 

22.8 

Maintenance  Test  Flights 

7.7 

10.4 

31.7 

21.8 

28.4 

Aircraft/Flightline  Operations 

9.6 

7.3 

3.5 

16.4 

32.5 

Aeromedical  Management 

11.2 

7.7 

38.3 

12.2 

30.5 

Crash,  Rescue,  and  Firefighting 

3.6 

4.2 

35.9 

21.8 

34.3 

Petroleum,  Oil,  and  Lubricants 

2.1 

4.0 

17.4 

19.7 

56.8 

Maintenance  Management  Training 

5.8 

8.2 

35.8 

19.8 

30.2 

Maintenance  Quality  Control 

4.6 

8.8 

26.5 

21.5 

38.8 

Maintenance  Shop  Operations 

1.7 

5.1 

33.7 

26.7 

32.6 

Aviation  Logistics 

12.1 

19.0 

34.2 

17.9 

16.8 

Aviation  Armament 

10.9 

6.5 

24.1 

17.9 

40.5 

Aviation  Life  Support  Equipment 

1.9 

3.1 

20.5 

17.9 

56.6 

Physical  Security 

8.3 

13.7 

35.2 

20.9 

22.0 

Across  Functional  Areas 

6.8 

8.9 

31.0 

20.0 

33.3 
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Table  6 


Criticality  Scale  Response  Percentages 


Response  Category 

Functional  Area  1  2  3  4  5 


Aviation  Safety  Management 
Facility/Unit  Operations 
Standardization  and  Training 
Maintenance  Test  Flights 
Aircraft/Flightline  Operations 
Aeromedical  Management 
Crash,  Rescue,  and  Firefighting 
Petroleum,  Oil,  and  Lubricants 
Maintenance  Management  Training 
Maintenance  Quality  Control 
Maintenance  Shop  Operations 
Aviation  Logistics 
Aviation  Armament 
Aviation  Life  Support  Equipment 
Physical  Security 

Across  Functional  Areas 


71.6 

15.0 

10.5 

2.0 

1.1 

34.5 

24.7 

23.9 

8.7 

8.2 

35.9 

24.3 

21.1 

9.5 

9.2 

41.9 

18.6 

23.1 

9.0 

7.3 

53.5 

11.3 

26.1 

4.2 

4.7 

55.9 

15.7 

18.5 

3.7 

6.0 

46.1 

25.3 

19.8 

4.6 

4.0 

35.2 

17.3 

21.7 

7.2 

18.6 

26.9 

24.6 

31.7 

8.4 

8.4 

34.2 

22.4 

24.1 

12.7 

6.8 

62.7 

20.3 

12.1 

1.8 

3.0 

50.8 

21.4 

20.4 

4.5 

2.9 

21.8 

31.2 

32.7 

5.1 

8.9 

32.9 

14.8 

28.6 

12.4 

11.3 

61.3 

21.1 

11.8 

5.0 

0.8 

44.3 

20.5 

21.7 

6.6 

6.8 
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PROCEDURE  FOR  REVISING  THE  ARMS  CHECKLIST 


The  results  from  this  research  provide  useful  information  to 
Army  decision  makers  about  the  Detectability,  Importance,  and 
Criticality  of  each  ARMS  Checklist  item  and  functional  area. 

This  section  of  the  report  recommends  a  three-step  procedure  for 
using  the  Detectability,  Importance,  and  Criticality  information 
to  decide  whether  to  retain,  revise,  or  delete  a  checklist  item. 
The  three  steps  are  summarized  in  the  flowchart  presented  in 
Figure  8  and  are  described  in  detail  in  the  following  paragraphs. 


Figure  8.  Flowchart  showing  recommended  procedure  for  revising 
the  ARMS  Checklist. 

Step  It  Establish  a  Hiah-Low  Cutoff  Point  for  Each  Scale 


The  first  step  requires  the  military  decision  maker  to 
establish  a  high-low  cutoff  point  for  each  of  the  three  scales. 
The  user  is  reminded  that  the  direction  of  the  original 
Detectability  and  Criticality  scales  was  reversed  prior  to  the 
data  analyses.  Therefore,  each  of  the  three  scales  progresses 
from  low  ratings  at  the  left  extreme  to  high  ratings  at  the  right 
extreme,  as  shown  in  Figures  2  through  7.  Somewhere  along  each 
scale  the  user  must  establish  a  cutoff  point  dividing  the  low 
response  categories  from  the  high  response  categories.  High 
response  categories  (above  the  cutoff  point)  describe 
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significant  deficiencies  that  should  be  examined  during  an  AKMS 
visit,  and  low  response  categories  (below  the  cutoff  point) 
describe  deficiencies  that  should  not  be  examined. 

Each  of  the  three  rating  scales  has  five  response 
categories  and  three  verbal  anchors.  The  verbal  anchors  for 
each  scale  will  assist  the  user  in  establishing  the  high-low 
cutoff  point.  The  portion  of  the  scale  that  describes 
significant  deficiencies  may  be  different  for  the  three  scales. 
In  fact,  the  verbal  anchors  and  the  data  shown  in  Figures  2 
through  7  suggest  that  a  different  cutoff  point  should  be 
established  for  the  Importance  and  Detectability  scales  than  for 
the  Criticality  scale.  Although  there  are  substantial 
differences  in  response  percentages  between  the  individual 
functional  areas  for  all  three  scales,  the  shape  of  the 
distributions  is  similar  for  each  scale  (see  Tables  4  through 
6) .  This  suggests  that  the  same  scale  cutoff  point  may  be  used 
for  all  functional  areas. 

As  an  example,  the  military  user  may  decide  that  a  facility 
must  be  able  to  support  at  least  75%  of  unit  training,  and  that 
a  unit  must  be  able  to  perform  at  least  75%  of  its  mobilization 
tasks.  According  to  the  verbal  anchors  on  the  Criticality  scale 
shown  in  Figures  6  and  7,  the  75%  cutoff  point  is  between  the 
first  and  second  response  categories.  Therefore,  response 
categories  2  through  5  on  the  Criticality  scale  describe 
significant  deficiencies.  The  military  user  may  also  decide 
that,  considering  the  limited  time  available  and  the  number  of 
potential  deficiencies  to  be  evaluated  during  an  ARMS  visit, 
checklist  item  deficiencies  should  be  more  than  moderately 
detectable  and  should  be  given  more  than  a  moderatt  amount  of 
weight.  In  these  cases,  according  to  the  verbal  anchors  shown 
in  Figures  2  through  5,  rating  categories  4  and  5  on  both  the 
Detectability  and  Importance  scales  would  describe  significant 
deficiencies  that  should  be  evaluated. 

Step  2;  Establish  Minimum  Response  Percentages 

The  second  step  requires  the  user  to  establish  the  minimum 
response  percentage  that  will  determine  if  an  item  is  above  the 
scale  cutoff  point  established  in  Step  1.  Items  with  response 
percentages  equal  to  or  greater  than  the  minimum  percentage  will 
be  considered  high  on  the  scale  of  interest.  Checklist  items 
with  response  percentages  lower  than  the  minimum  percentage  will 
be  considered  low  on  the  scale  of  interest. 

The  response  percentages  presented  in  Tables  4  through  6 
provide  empirical  data  for  determining  the  minimum  response 
percentages  for  each  scale.  Establishment  of  the  minimum 
percentage  is  illustrated  using  the  high-low  cutoff  points 
described  in  the  example  for  Step  1.  Across  all  items, 
approximately  55%  of  the  Criticality  responses  occur  in  rating 
categories  2  through  5.  Approximately  60%  of  the  Detectability 
responses  and  53%  of  the  Importance  responses  occur  in  rating 
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categories  4  and  5.  These  data  can  be  used  to  identify  response 
percentages  for  checklist  items  that  are  substantially  different 
from  the  expected  percentages. 

Cohen  (1977)  suggested  that,  for  categorical  data,  a 
percentage  that  is  10%  greater  than  the  expected  percentage 
would  probably  be  statistically  significant  for  relatively 
small  samples.  Applying  this  percentage  to  the  example, 
checklist  items  with  65%  or  more  of  the  responses  in  categories 
2  through  5  on  the  Criticality  scale  would  be  considered  to  have 
a  high  Criticality  score.  Checklist  items  with  70%  or  more  of 
the  responses  in  categories  4  and  5  on  the  Detectability  scale 
would  be  considered  to  have  a  high  Detectability  score. 

Likewise,  items  with  63%  or  more  of  the  responses  in  categories  4 
and  5  on  the  Importance  scale  would  be  considered  to  have  a  high 
Importance  score. 

The  Step  2  recommendations  are  presented  as  general 
guidelines;  the  military  user  should  establish  criteria  that  are 
considered  to  be  meaningful  and  useful.  For  example,  a  higher 
adjustment  than  Cohen's  recommended  10%  (e.g.,  15  or  20%)  above 
the  expected  response  percentage  could  be  imposed  if  the 
evaluators  decided  to  set  more  stringent  criteria  for  the  core 
functional  areas. 

S-t^gp  J? ; _ Exercise  Decision  Rulejs 

Once  the  criteria  for  determining  whether  a  checklist  item 
is  high  or  low  on  each  scale  are  established,  the  next  step  is 
to  decide  whether  to  retain,  revise,  or  delete  each  item.  A 
recommended  set  of  decision  rules  for  combining  the  ratings  on 
the  three  scales  is  presented  in  Figure  9.  The  decision  rules 
should  be  applied  to  both  the  facility  ratings  and  the  unit 
ratings. 

The  decision  process  begins  with  the  decision  about  the 
checklist  item's  high-low  rating  on  Criticality.  In  the 
decision  flowchart,  if  the  Criticality  rating  is  high,  the  user 
will  proceed  to  the  right;  if  the  rating  is  not  high  (i.e., 
low) ,  the  user  will  proceed  to  the  left  and  downward. 
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If  the  item's  Criticality,  Importance,  and  Detectability 
ratings  are  all  high,  the  item  should  be  retained  in  the 
checklist  unchanged.  If  only  the  Detectability  rating  is  low, 
an  attempt  should  be  made  to  revise  the  item  to  improve  its 
Detectability  (i.e.,  state  the  item  more  specifically  or  divide 
it  into  more  than  one  item) .  If  the  item  has  a  high 
Criticality  rating,  a  low  Importance  rating,  and  a  high 
Detectability  rating,  it  probably  is  assessing  a  deficiency  in 
a  different  functional  area.  Such  an  item  should  be  reassigned 
to  a  more  appropriate  functional  area.  If  the  item  has  a  nigh 
Criticality  rating  but  low  Importance  and  Detectability 
ratings,  it  should  be  revised  and  reassigned  to  another 
functional  area. 

If  the  item's  Criticality  rating  is  low  and  the  Importance 
rating  also  is  low,  it  may  be  advisable  to  delete  or  revise  the 
item,  depending  on  whether: 

-  the  item  is  required  by  a  current  regulation, 

-  the  subject  matter  is  covered  in  another  item,  or 

-  the  item  has  a  low  Detectability  rating. 

If  the  item  has  a  low  Criticality  rating,  a  high  Importance 
rating,  and  a  high  Detectability  rating,  it  should  be  retained 
in  the  checklist  unchanged.  If  the  item  has  a  low  Criticality 
rating,  a  high  Importance  rating,  and  a  low  Detectability 
rating,  it  should  be  revised  to  make  the  deficiency  easier  to 
detect . 

The  examples  described  above  are  not  exhaustive.  Rather, 
the  flowchart  is  intended  to  provide  a  general  framework  for 
military  decision  makers;  additional  factors  may  need  to  be 
considered  when  deciding  to  retain,  revise,  or  delete  a 
checklist  item.  In  addition,  items  in  some  functional  areas 
may  have  low  Detectability,  Importance,  or  Criticality  ratings 
for  a  facility,  but  have  high  Detectability,  Importance,  and 
Criticality  ratings  for  a  unit,  or  vice  versa.  Some  items  may 
be  deleted  for  either  a  facility  or  a  unit  but  retained  or 
revised  for  the  other. 

In  summary,  items  whose  Detectability,  Importance,  and 
Criticality  ratings  for  a  facility  or  a  unit  are  all  high  most 
likely  should  be  retained  in  their  present  form.  The  user 
should  consider  deleting  items  with  low  ratings  on  all  three 
scales  unless  one  of  the  conditions  illustrated  in  the  flowchart 
shown  in  Figure  9  is  met.  The  flowchart  also  provides  decision 
rules  for  various  combinations  of  low  or  high  ratings  on  the 
three  scales. 
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DISCUSSION 


This  section  of  the  report  summarizes  and  discusses  the 
results  of  the  AKMS  Checklist  evaluation.  The  section  is 
divided  into  two  parts.  The  first  part  discusses  the 
implications  of  the  checklist  item  rating  scale  results.  The 
second  part  discusses  the  purpose  and  uses  of  the  ARMS 
Checklist  Data  Base. 

The  data  summarized  in  this  report  reflect  the  judgments 
of  RC  aviators  and  nonrated  aviation  personnel  who  are 
responsible  for  managing  one  or  more  of  the  functional  areas 
in  a  facility  or  a  unit.  The  research  does  not  constitute  a 
definitive  evaluation  of  the  ARMS  Checklist  because  ratings 
were  not  available  from  other  aviation  SMEs;  however,  the 
results  provide  useful  guidance  for  Improving  both  the  content 
of  the  checklist  and  the  procedures  used  to  evaluate  RC 
facilities  and  units. 

Checklist  Item  Rating  Scale  Results 

On  the  average,  only  7%  of  the  responses  were  in  the 
lowest  rating  category  on  either  the  Detectability  or  the 
Importance  scales.  This  finding  suggests  that  during  an  ARMS 
evaluation  visit: 

-  it  would  be  easy  to  detect  the  majority  of  the 
deficiencies  described  in  the  checklist  items,  and 

~  the  majority  of  deficiencies  should  be  given  at  least 
a  moderate  amount  of  weight. 

On  the  average,  45%  of  the  responses  were  in  the  lowest  rating 
category  on  the  Criticality  scale,  suggesting  that,  in  general: 

-  a  facility  with  the  deficiency  described  in  an  item 
could  support  most  aspects  of  unit  training,  and 

-  a  unit  with  the  deficiency  described  in  an  item  could 
perform  most  of  its  mobilization  tasks. 

These  findings  should  be  interpreted  with  caution  for  the 
following  three  reasons.  First,  the  respondents  were  instructed 
to  rate  the  Criticality,  as  well  as  the  Detectability  and 
Importance,  of  each  item  as  if  the  deficiency  were  the  only 
deficiency  that  existed  in  a  facility  or  in  a  unit.  It  may  be 
argued  that  few  of  the  deficiencies  described  in  the  items,  in 
isolation,  would  either  (a)  prevent  a  facility  from  supporting 
a  unit's  training,  or  (b)  prevent  a  unit  from  performing  its 
mobilization  tasks.  Even  when  several  deficiencies  exist 
simultaneously,  they  may  not  prevent  the  accomplishment  of  the 
facility  or  unit  mission.  Unfortunately,  it  is  impossible  to 
conclude  from  the  data  what  the  effect  of  different  combinations 
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of  deficiencies  night  be,  or  to  stinate  a  facility's  or  unit's 
capability  to  overcome  such  deficiencies. 

Second,  even  though  approximately  45%  of  the  responses 
were  in  the  lowest  Criticality  rating  category  for  the  entire 
checklist,  the  percentages  vary  substantially  between  the 
functional  areas.  The  data  suggest  that,  on  the  whole,  the 
deficiencies  described  by  the  items  in  certain  areas  (e.g.. 
Aviation  Safety  Management,  Physical  Security)  nay  be  somewhat 
less  critical  to  mission  accomplishment  than  the  deficiencies 
described  by  the  items  in  other  areas  (e.g..  Aviation  Armament, 
Maintenance  Management  Training) .  These  differences  between 
functional  areas  are  easily  identified  by  examining  Tables  4 
through  6 . 

Third,  the  rating  distributions  in  Figures  2  through  7 
and  the  rating  scale  verbal  anchors  suggest  that  different 
criteria  may  be  appropriate  for  the  Detectability,  Importance 
and  Criticality  scales.  For  example,  to  retain  an  item  in  the 
checklist,  the  user  may  require  that  (a)  70%  or  more  of  the 
responses  be  in  categories  4  and  5  on  the  Detectability  scale, 
(b)  63%  or  more  of  the  responses  be  in  categories  4  and  5  on 
the  Importance  scale,  and  (c)  65%  or  more  of  the  responses  be 
in  categories  2  through  5  on  the  Criticality  scale. 

In  general,  the  data  indicate  that  the  items  received 
similar  ratings  for  a  facility  and  for  a  unit.  This  suggests 
that  developing  one  checklist  to  use  when  evaluating  a  facility 
and  another  checklist  to  use  when  evaluating  a  unit  probably  is 
not  necessary.  Instead,  a  single  checklist  should  be  developed, 
with  the  items  that  apply  only  to  a  facility  or  only  to  a  unit 
clearly  identified.  The  decision  flowchart  presented  in  Figure 
9  and  described  in  the  previous  section  provides  a  set*  of 
recommended  decision  rules  for  accomplishing  this.  The 
flowchart  should  be  applied  separately  to  the  facility  ratings 
and  to  the  unit  ratings. 

As  described  previously,  data  are  provided  in  this  report 
for  each  item  and  for  the  group  of  items  used  to  assess  each 
functional  area.  The  two  types  of  data  are  intended  to  serve 
two  different  functions.  The  summary  data  for  each  of  the 
functional  areas  shown  in  Tables  4  through  6  should  be  used  to 
establish  an  operationally  significant  criterion  for  each  of 
the  three  scales  and  to  identify  functional  areas  whose  items 
have  lower  average  Detectability,  Importance,  and  Criticality 
ratings.  The  data  for  the  individual  items  presented  in  Volume 
II  (Appendix  C)  should  be  considered  when  making  decisions 
about  retaining,  revising,  or  deleting  specific  items.  Neither 
type  of  summary  data,  however,  provides  the  basis  for 
determining  if  additional  items  are  required  to  make  the  ARMS 
evaluation  comprehensive. 
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ARMS  Checklist  Data  Base 


Information  contained  in  the  ARMS  Checklist  Data  Base  can 
be  used  to  summarize  (a)  the  ARNG  and  USAR  aviators'  ratings  of 
the  Detectability,  Importance,  and  Criticality  of  the  checklist 
items,  and  (b)  the  performance  of  ARNG  and  USAR  units  on 
specific  checklist  items  and  functional  areas  during  future 
ARMS  visits.  Even  before  additional  data  are  collected,  the 
data  base  information  will  allow  the  First  Army  ARMS  team  to 
reorganize  the  checklist  by  identifying  items  with  similar 
content  and  reference  publications.  In  addition,  the 
information  can  be  used  to  generate  reports,  to  identify 
recurring  specific  unit  and  facility  strengths  and  weaknesses, 
to  identify  common  areas  of  strengths  and  weaknesses,  or  to 
process  additional  data  collected  for  future  revisions  of  the 
checklist. 
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RECOMMENDATIONS 


This  section  presents  nine  recomnendations  for  improving 
the  ARMS  Checklist  and  evaluation  procedure.  The 
recommendations  are  drawn  from  the  results  of  the  rating  data 
analysis  described  in  this  report,  observations  made  by  project 
personnel  during  ARMS  evaluation  visits,  and  discussions  with 
ARMS  team  members.  The  first  five  recommendations  suggest 
changes  in  the  content  of  the  checklist: 

-  The  decision  to  retain  an  item  in  its  present  form, 
revise  the  item,  or  delete  it  from  the  checklist 
should  be  based  on  an  assessment  of  the  item's 
Criticality,  Importance,  and  Detectability  ratings 
for  both  a  facility  and  a  unit. 

Items  with  high  overall  Detectability,  Importance, 
and  Criticality  ratings  should  be  retained  in  the 
checklist  in  their  present  form. 

Items  with  low  overall  ratings  on  only  one  or  two  of 
the  three  scales  should  be  revised  or  reassigned  to 
another  functional  area  according  to  the  decision 
rules  summarized  in  Figure  9. 

-  Items  with  low  overall  Detectability,  Importance, 
and  Criticality  ratings  probably  should  be  deleted 
from  the  checklist  unless  one  of  the  conditions 
illustrated  in  Figure  9  is  met. 

A  single  version  of  the  checklist  should  be  used, 
rather  than  separate  versions  for  a  facility  and  a 
unit.  Items  that  pertain  only  to  a  facility  or  only 
to  a  unit  should  be  identified  on  the  single 
checklist,  as  illustrated  in  Figure  10. 

The  next  two  recommendations  suggest  changes  to  the 
organization  and  format  of  the  checklist: 

-  Each  page  of  the  checklist  should  contain  a  heading 
that  identifies  the  functional  area  and  the 
functional  subarea  of  the  items,  as  shown  in  Figure 
10. 


The  ARMS  Checklist  should  be  revised  so  that  the 
items  are  stated  positively  as  questions  (see  Figure 
10)  rather  than  negatively  as  deficiencies  (cf. 
Carpenter  and  Just,  1975;  Wickens,  1984).  The  new 
question-format  items  can  be  linked  to  the  old 
deficiency-format  items  by  means  of  a  word 
processing  program.  In  this  manner,  the 
deficiency- format  can  be  retained  if  necessary  for 
informal  feedback  and  formal  reports. 
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AVIATION  RESOURCE  MANAGEMENT  SURVEY 


Standardization  and  Training:  Aircrew  Training  Program 


Facility/ 
Item  Unit 


Number 

Only 

Item  Name 

229/X 

Has  a  terrain  flight  training  area  been 
designated? 

229/? 

'• 

Has  an  individual  night  tactical  training 
program  been  established? 

229/$ 

F 

Is  there  an  Individual  Aircrew  Training 

Folder  for  each  aviator? 

229/+ 

Have  all  aviators  been  evaluated  and  placed 
in  an  appropriate  Flight  Activity  Category? 

230/q 

U 

Has  the  Command  established  a  mission 
training  program? 

230/y 

Does  dociimentation  exist  to  indicate  that  new 
aviators  were  receiving  local  area 
orientations? 

230/G 

Has  the  Annual  Written  Examination  been 
administered  to  all  aviators? 

Figure  10.  Illustration  of  recommended  changes  to  the 
checklist  format. 


The  last  two  recommendations  suggest  potential  applications 
of  the  ARMS  Checklist  Data  Base: 

The  ARMS  Checklist  Data  Base  should  be  used  as  an 
aid  for  further  refinements  to  the  checklist  format. 
For  example,  the  data  base  can  be  used  to  identify 
items  that  deal  with  the  same  subject  matter  or  that 
refer  to  the  same  publications.  It  may  be  more 
efficient  to  group  these  items.  In  addition,  this 
information  may  be  useful  for  determining  the 
sequence  of  items  that  will  minimize  evaluator 
effort. 
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The  ARMS  Checklist  Data  Base  should  be  used  for 
several  different  analyses.  Additional  data  fields 
can  be  created  to  record  the  facilities  and  units 
having  observed  deficiencies.  This  will  help 
identify  conunonly  occurring  deficiencies,  facilitate 
the  preparation  of  annual  susa&ary  reports,  and 
increase  the  quality  of  feedback  to  command 
personnel . 
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Part  A 
General 

1.  Purpose:  This  pamphlet  has  been  prepared  for  use  In  the  conduct  of 
Aviation  Standardization  and  Training  Program  Evaluations  and  Aviation 
Resource  Hanagetsent  Surveys  performed  on  Reserve  Component  Ccraands  and  their 
organic,  assigned  or  attached  aviation  assets.  The  purpose  of  the  guide  is  to 
provide  a  uniform  format  for  conducting  and  reporting  results  of  evaluations/ 
inspections/surveys/visits  performed  in  accordance  with  paragraphs  3-13  and 
3-20,  FORSCOM  Reg  350-3  and  AR  385-95  with  applicable  supplements.  The  guide 
does  not  replace  the  applicable  edition  of  the  US  Army  Safety  Center  Guide  to 
Aviation  Resources  Kanagement  for  Aircraft  Mishap  Prevention,  established  as 
the  PORSCOH  ARKS  standard  publication,  but  augments  the  guide  to  facilitate 
timely  report  preparation  in  a  uniform  format.  It  is  not  suggested  that  the 
guide  be  used  as  a  single  source  document  in  preparing  for  surveys  and 
evaluations.  US  Army  Safety  Center  Guide  to  Aviation  Resources  Management  for 
Aircraft  Mishap  Prevention,  with  local  references  update  and  other  applicable 
HQDA,  KGB,  PORSCOK  and  First  Army  publications  are  reconinended  as  preparation 
guides. 

2.  Organization:  The  guide  contains  significant  items  that  impact  on  Reserve 
Component  Aviation  elements*  safety  and  readiness  or  have  been  designated  as 
special  subjects  for  evaluation  by  FORSCOM  or  higher  headquarters.  First  Army 
identified  special  subjects  apply  to  USAR  only.  The  number  of  comnents/pages 
that  are  used  to  cover  a  specific  area  or  subarea  should  not  be  used  to 
determine  the  Importance  of  any  given  area. 

3.  Bvaluation/Survey  Ratings.  Standards.  Briefings,  Reports,  and  Corrective 
Action  Plan: 


a.  Ratings;  An  aviation  element  will  receive  one  of  two  overall  ratings, 
satisfactory  or  unsatisfactory,  for  an  evaluation  or  a  survey.  Major  func¬ 
tional  areas  will  also  be  rated  satisfactory  or  unsatisfactory.  Subareas 
which  will  appear  in  the  report  may  also  be  rated  unsatisfactory;  however,  the 
absences  of  the  words  satisfactory  or  unsatisfactory  mean  the  subarea  was 
awarded  a  satisfactory  rating.  The  evaluation  of  the  Army  Aviation  Support 
Facilities  and  Army  Aviation  Plight  Activities  satisfies  the  Inspector  General 
inspection  requirement  lAW  lA  Reg  350-15. 

b.  Standards:  The  overall  rating  for  an  evaluation  or  a  survey  will  be 
based  upon  the  relative  significance  of  functional  areas  to  overall  safety 
practices,  directives  compliance  and  tralnlng/readlness .  All  12  functional 
areas  listed  in  Part  B  are  not  applicable  to  all  elements;  therefore,  only 
applicable  areas  will  be  evaluated  and  rated.  Six  principal  functional  areas 
apply  to  all  elements  subject  to  evaluations  or  surveys  -  Safety  Management, 
Facillty/Unit  Operations,  Standardization  and  Training,  Aircraft  Operations, 
Maintenance  Manageotent,  and  Aviation  Life  Support  Equipment.  Normally, 
unsatisfactory  ratings  on  any  two  of  the  principal  areas  will  constitute  an 
overall  unsatisfactory.  An  overall  unsatisfactory  may  be  awarded  when  any  one 
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of  six  principal  areas  and  any  two  of  the  other  six  areas  are  rated  unsatis- 
fac^’ory.  Because  the  type  aviation  elements  and  situations  confronting  these 
elements  vary  throughout  First  US  Army,  these  standards  are  somewhat  flexible 
and  will  be  applied  with  judgment. 

c.  Briefings:  Chief,  ARMS  evaluation  team  will  give  an  entrance  briefing. 

The  purpose  Is  to  outline  the  evaluation,  ratings,  outbriefing,  report,  and 

corrective  plan  procedures  to  the  commanders/supervisors  of  the  facilities, 
activities,  and  units  and  their  staffs.  After  the  completion  of  each  evalua¬ 
tion,  the  team  will  conduct  an  in-depth  exit  briefing  for  the  facilities, 
activities,  and  units.-  Upon  the  conclusion  of  the  ARMS  evaluations  within  a 
Major  US  Army  Reserve  Command  or  State,  the  ARMS  team  will  conduct  a  formal 
exit  briefing  for  the  Commander  of  the  MUSARC  or  the  Adjutant  General  of  the 
State  or  his  designated  representative. 

d.  Reports :  Upon  conclusion  of  the  ARMS  evaluation,  an  unofficial  list 
of  the  discrepancies/findings,  grades! ips,  and  observation  sheets  will  be 
provided  to  the  commanders/su^rvisors  of  the  facilities,  activities,  and 
units.  This  will  be  done  during  the  outbriefing.  A  formal  written  report 
will  be  prepared  and  distributed  within  60  days  of  the  evaluation.  It  will 
provide  a  summary  of  the  evaluation's  ratings,  commendable  areas,  and  subjects 
of  concern.  The  report  will  include  separate  enclusjres  for  each  facility  and 
unit  that  were  evaluated.  Enclosures  will  identify  «.'ach  discrepancy  with 
applicable  referenceis ),  flight  evaluation  gradeslips,  and  observation 
sheets.  Areas  within  each  enclosure  requiring  corrective  action  will  be 
designated  with  an  asterisk. 

e.  Corrective  Action  Plan;  The  formal  written  report  will  designate 
speci fic  discrepancies  wi  th  asterisks.  The  address  will  submit  a  corrective 
plan  to  this  Headquarters,  ATTN;  AFKA-TR-A,  by  the  designated  suspense  date 
within  the  report.  The  Aviation  Division  will  review  the  corrective  action 
plan  and  will  initiate  assistance  action  through  the  Centralized  Aviation 
Readiness  Training  Team  and  staffs  within  First  US  Army,  US  Army  Forces 
Command,  and  other  Army  agencies. 

4.  Evaluation/Survey  Comments:  Coding  of  comments  within  the  guide  facili- 
tate  processing  of  reports.  Following  is  an  explanation  of  the  coding  system: 

44  a _  Word  Processing  Equipment  Comment  Index  Nur.  jer. 

_  Word  Processing  Equipment  Glossary  Code. 

G _  R _  Indicates  applicability  to  Army  National  Guard, 

_  Indicates  applicability  to  Army  Reserve. 


5,  Proponent:  The  proponent  agency  of  the  pamphlet  is  Aviation  Division, 
Office  of  the  Deputy  Chief  of  Staff,  Training,  Headquarters,  First  US  Army. 
Questions  regarding  listed  comments  in  the  guide  should  be  addressed  to  the 
proponent.  Users  are  invited  to  send  comments  and  suggested  improvements  on 
DA  Form  2028  (Recommended  Changes  to  Publications  and  Blank  Forms)  direct  to 
Commander,  First  US  Army,  ATTN:  AFKA-TR-A,  Fort  George  G.  Meade,  Maryland 
20755. 
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Part  B 


Reserve  Components  Commemder' s  Guide 
Aviation  Standardization  and  Training  Program  Evaluation 

and 

Aviation  Resource  Management  Survey  <.  « 


REMARK  NO. 


REHAJUCS 


.APPLIES  TO 
226/a 
226/b 
226/c 
226/d  G 
226/e  R 


I.  AVIATION  SAFETY  MANAGEMENT: _ 

AVIATION  SAFETY  OFFICER  (ASO) 

An  ASO  had  not  been  authorized/assigned/appointed. 

Recommend  compliance  with  para  I-4d,  NGR  385-10  and  para  1-6d,  AR  385-95. 

Recommend  compliance  with  para  l-6d,  AR  385-95  and  FORSCOM/TRAOOC  Suppl  1 
to  AR  385-95. 


22  6/ f 
226/g  G 

226/h  R 

226/ i 
22  6/ j 

226/k 

226/1 

226/m 

226/n 

226/0 

226/p 

225/q 


ASO  was  not  school  trained. 

Recommend  compliance  with  para  l-4c(6),  NGR  385-10,  and  AR  385-95, 
paragraphs  l-6d  and  1-7a(ll). 

Recommend  compliance  with  para  l-6d  and  1-7a(n)(d),  AR  385-95  and 
FORSCOM/TRAOOC  Suppl  1  to  AR  385-95. 

ASO  had  not  performed  or  documented  the  following  duties  as  appropriate: 

Observed  Flight  and  ground  operations  to  detect  and  correct  unsafe 
practices. 

Conduct  hazard  analysis,  rank  hazards  in  terms  of  severity  and  accident 
probability,  and  advised  responsible  officials  promptly.  . 

Educate  aircrew  members  on  safety  related  subjects. 

Review  aircraft  accident  reports  and  help  implement  corrections. 

Rehearse  and  review  adequacy  of  the  preaccident  plan. 

Ensure  that  communication  equipment,  navigation  aids,  and  other  electronic 
aids  to  aircraft  operations  are  inspected. 

Inspect  physical  condition  of  air  fiel  ds ,  heliports,  and  tactical  landing 
sites  for  hazards;  recommend  improvements;  and  insure  that  all  knc^n 
hazards  are  publicized. 

Maintain  current  reference  files  of  aviation  safety  literature. 


22  6/ r 


Maintain  organizational  aircraft  accident  records. 


226/s 


Review  aviation  flight  records  and  the  unit  training  program  and  make 
recommendations  to  correct  deficiencies. 
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remark  mo. 


APPLIES  TO 
226/t 

226/u 

226/v 


REMARXS 


226/; 


0009/b 


Advise  all  aviators  on  safety  and  the  importance  of  following  star 
procedures  and  techniques. 

I  Monitor  techniques  and  proficiency  of  personnel  in  handling  weapons, 
ammunition;  and  petroleum,  oil  and  lubricants  (POL). 

Observe  aviation  maintenance  operations  and  make  recommendations  to 
correct  unsafe  procedures  and  practices. 

Manage  Operational  Hazard  Report  (OHR)  functions. 

Monitor  the  FOD  prevention  functions. 

Advise  and  assist  aircraft  accident  investigation  boards. 

Analyze  accidents  and  results  of  accident  prevention  surveys. 

Monitor  aviation  lif2  support  equipment  (ALSE)  and  related  survival 
training  programs. 

Take  part  in  mission  planning  to  insure  weather,  terrain,  areas  of 
operation,  and  crew  and  aircraft  capabilities  are  considered. 

Perform  other  duties  as  outlined  in  DA  Pam  385-95. 

Recommend  compliance  with  para  l-5b(4),  NCR  385-10  and  para  l-7c, 

AR  385-95. 

Recommend  compliance  with  para  l-7c,  AR  385-95. 

AVIATION  SAFETY  COUNCILS/MEETINGS 
An  aviation  safety  council  had  not  been  established. 

Recommend  compliance  with  para  l-6a,  NCR  385-10  and  para  5-2,  NCR  95-1. 
Recommend  compliance  with  para  2-7,  AR  385-95. 

Aviation  safety  council  did  not  include  appropriate  membership. 

Recommend  compliance  with  para  l-6a,  NCR  385-10  and  para  2-7,  AR  385-95. 
Recommend  compliance  with  para  2-7,  AR  385-95. 

Command  Safety  and  Occupational  Health  Advisory  Council  Committee  did  no< 
have  at  least  one  aviation  representative. 

Recotvnend  compliance  with  para  l-4d(3),  NCR  385-10. 


0009/c  R 


Recommend  compliance  with  para  2-7e,  FORSCOM/TRADOC  Suppl  1  to  AR  385-95 
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ABRIDGED  VERSION  OF  THE  ARMS  CHECKLIST 
RATING  BOOKLET  FOR  THE  AIRCRAFT/ FLIGHTLINE  OPERATIONS 

FUNCTIONAL  AREA 


AVIATION  RESOURCE  MANAGEMENT  SURVEY 
(ARMS)  CHECKLIST  ITEM  QUESTIONNAIRE 


During  an  Aviation  Resource  Management  Sun/ey  (ARMS),  several  functional  areas  (e.g. 
aviation  safety  management,  standardization  and  training,  maintenance  management)  that  are 
aitical  to  the  mission  of  an  aviation  support  facility  and  of  a  Reserve  Component  (RC)  aviation 
unit  are  evaluated  by  an  ARMS  team.  A  number  of  Items  that  identify  specific  deficiencies 
are  used  as  evaluative  guides  by  the  ARMS  team  members.  Information  from  these  items  is 
used  to  assign  an  overall  rating  of  Satisfactory  or  Unsatisfactory  to  the  facility  and  units 
evaluated.  Constructive  feedback  based  on  the  findings  from  the  ARMS  is,  in  turn,  provided  to 
a  fadlity  or  to  a  unit  about  its  strengths  and  weaknesses. 

The  First  U.  S.  Army  Deputy  Chief  of  Staff  for  Training  (DCST),  Aviation  Divison,  has 
requested  that  the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
conduct  research  to  evaluate  the  checklist  and  procedure  used  by  the  First  Army  ARMS  team, 
to  identify  problems  that  may  exist,  and  to  recommend  ways  in  which  the  checklist  and 
procedure  can  be  improved.  This  questonnaire  is  part  of  that  effort.  The  purpose  of  the 
questionnaire  is  to  obtain  the  judgments  of  subject  matter  experts  such  as  yourself  about 
certain  characteristics  of  the  checklist  items.  *nie  research  findings  will  be  used  to  improve 
the  ARMS  and  to  provide  operationally  useful  feedback  to  Reserve  Component  command 
personnel,  and  will  be  valuable  to  ARMS  teams  from  all  the  Continental  U.  S.  Annies. 

On  page  5  of  the  questionnaire,  you  will  be  asked  to  provide  some  general  demographic 
information.  This  information  will  help  ARI  to  understand  the  rating  data  you  provide.  On 
page  6  you  will  find  definitions  of  three  characteristics  of  the  checklist  items.  Read  the 
definitions  carefully,  and  refer  to  them  as  necessary.  Two  sample  items  are  shown  on  page  7 
and  on  page  8.  Please  review  the  examples  before  beginning  the  rating  task.  Page  9  contains  a 
brief  explanation  of  the  two  examples. 

Located  at  the  top  of  each  of  the  remsuning  pages  in  the  questionnaire  is  an  item  that  is 
currently  used  in  the  First  Army  ARMS  checklist,  and  the  functional  area  from  which  the  item 
is  taken.  Each  item  describes  a  deficiency  that  may  be  found  in  a  support  facility  or  in  a  unit. 
Below  the  item  are  two  sets  of  rating  items  -  rating  items  1 , 2,  and  3  are  to  be  used  to  rate  a 
facility;  rating  items  4,  5,  and  6  are  to  be  used  to  rate  a  unit. 

After  you  have  completed  the  questionnaire  booklet,  please  mail  it  to  ARI  at  the  following 
address: 


ARI  Aviation  Research  and  Development  Activity 
ATTN:  PERI-IR  (ARMS) 

Fort  Rucker,  Al^ama  36362'5354 

We  appreciate  your  cooperation.  Your  responses  will  be  confidential  and  will  be  used  for 
research  purposes  only. 


Please  turn  the  page  and  begin. 
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DATA  REQUIRED  BY  THE  PRIVACY  ACT  OF  1974 
IS  tiS  C  SSUmf 


TiTLroF fORM  Aviation  Kesource  nanagement  purvey 
Item  Questionnaire 


3.  RRiNClRAL  eUAROSCiS) 


The  data  collected  with  the  attached  questionnaire  are  to  be  used  for  research  purposes 
only. 


3.  nOUTINS  USES 


The  purpose  of  the  research  Is  to  evaluate  the  content  and  procedure  used  to 
conduct  the  Aviation  Resource  Management  Survey  (ARMS)  In  First  Army  Reserve 
Component  (RC)  facilities  and  units.  The  research  will  provide  Information  about 
(a)  critical  RC  facility  and  unit  deficiencies  that  should  be  evaluated  during  an 
ARMS  visit,  (b)  Improved  procedures  for  using  a  checklist  to  evaluate  RC 
facilities  and  units,  and  (c)  Improved  procedures  for  managing  and  Interpreting 
evaluative  information  from  ARMS  visits. 

When  an  Identifier  (e.g.,  Social  Security  Number)  Is  required,  it  Is  to  be  used 
for  administrative  and  statistical  control  purposes  within  the  confines  of  the 
subject  research.  Full  confidentiality  of  the  response  will  be  maintained. 


4  MANOATOnV  oa  VOLUNTAWY  DISCLOSuaE  *NO  Ef  FECT  ON  INO'VIOUAI.  NOT  FBOVIOINC  INFOBMATION 

Your  participation  In  Che  research  Is  strictly  voluntary.  You  are  encouraged  to 
provide  complete  and  accurate  information  in  the  interests  of  the  research,  but  there 
will  be  no  effect  on  you  for  not  providing  all,  or  any  part  of,  the  information. 

You  may  detach  this  notice  from  the  questionnaire  If  you  desire  to  do  so. 


•  Pn*acv  Act  Stattmtnt  -  26  Sap  75 


FT  CONTROL  NUMBER:  SfiSOA 


da  control  NUMBER: 


SUBJECT  MATTER  EXPERT  DEMOGRAPHIC  INFORMATION 


Check  [  >/  ]  the  appropriate  box  or  write  the  required  information  in  the  appropriate  space. 


1 .  What  are  the  last  four  digits  of  your  social  secutiry  number? 


2.  Which  of  the  following  best  desaibes  your  present  duty  position?  (check  only  one) 


Aviation  Resource  Management  Survey  (ARMS)  Team  Member 
Centralized  Aviation  Readiness  Training  (CART)  Team  Member 
Outside  Evaluation  Specialist  (e.g..  SIP,  MTFE) 

Full-time  Federal  Technician  Assigned  to  an  ARNG  Support  Facility 

Full-time  Federal  Technician  Assigned  to  a  USAR  Facility 

Member  of  an  ARNG  unit,  but  not  a  Full-time  Federal  Te^nidan  at  an  ARNG  Fadlity 

Member  of  a  USAR  unit  but  not  a  FuB-time  Federal  Technician  at  a  USAR  Facility 

Other  (specify) _ _ _ _ 


3.  How  long  have  you  been  in  your  present  duty  position?. 


Years 


and 


Months 


4.  Indicate  the  Continental  U.  S.  Army  (CONUSA)  to  which  you  are  assigned. 

[  1  1  First  [  2  1  Second  [  4  ]  Fourth  (  s  ]  Fifth  (  6  ]  Sixth 

5.  What  is  your  military  grade? 


1 

1 

1 

E-5 

(  6  1 

W-1 

[lol 

0-1 

[ 

2 

1 

E-6 

1  7  1 

W-2 

(111 

0-2 

[ 

3 

1 

E-7 

[  8  1 

W-3 

[12] 

0-3 

I 

4 

1 

E-8 

I  9  ) 

W-4 

{  13l 

0-4 

( 

5 

1 

E-9 

[I4l 

0-5 

(IS)  0-6 


6.  How  many  years/months  of  total  military  service  do  you  have? _ and _ 

Years  Months 


7.  What  is  your  total  number  of  military  flight  hours? _ Hours 

8.  In  which  of  the  following  functional  areas  do  you  consider  yourself  to  have  the  most 
knowledge  and  expertise?  (check  only  one) 


[  1 

]  Aeromedicai  Management 

(  7  1 

Aviation  Safety  Management 

[  2 

j  Aircraft  Crash  Rescue  and  Fire  Fighting 

1  e  1 

Aviation  Stancterdization  and  Training 

[  3 

1  Aircraft/Flight  Line  Operations 

1  9  1 

FacilityAJnit  Operations 

I  < 

]  ATC  Management/Training 

(  10  1 

Maintenance  Management 

(  3 

j  Aviation  Armament 

(  111 

POL  Facilities  and  Operations 

1  8 

j  Aviation  Life  Support  Equipment 

1  12  ) 

Physical  Security 

9.  Have  you  ever  served  in  an  aviation  position  in  the  Army  National  Guard  ? 

[  ol  No 

{  ij  Yes  •  If  Yes,  indicate  your  total  time:  _ and _ 

Years  Months 

10.  Have  you  ever  served  in  an  aviation  position  in  the  Army  Reserve  ? 

(  ol  No 

(  1  j  Yes  •  If  Yes,  indicate  your  total  time:  _ and _ 

Years  Months 


Please  turn  to  page  6  and  review  the  definitbns  of  the  rating  scales. 
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DEFINITIONS  OF  RATING  SCALES 

Listed  below  are  definitions  of  the  three  characteristics  of  the  checklist  items  that  you  will 
rate.  The  characteristics  are  defined  first,  as  they  apply  to  an  aviation  support  facility, 
and  second,  as  they  apply  to  a  Reserve  Component  aviation  unit.  Please  read  each 
definition  carefully  before  you  begin  the  rating  task.  Refer  to  the  definitions  as  often  as 
necessary. 


Support  Facility 


Item 

Detectability 


The  relative  ease  or  difficulty  of  determining,  during  an  ARMS 
evaluation,  if  the  deficiency  exists  in  a  support  facility 


Functional  Area 
Importance 


The  amount  of  weight  that  the  deficiency  should  be  given  when 
evaluating  Aviation  Standardization  and  Training  in  a  support 
facility 


Mission  The  extent  to  which  a  facility  with  the  deficiency  can 

Criticality  support  the  training  of  a  Reserve  Component  unit 


Reserve  Component  Unit 


Item 

Detectability 


The  relative  ease  or  difficulty  of  determining,  during  an  ARMS 
evaluation,  if  the  deficiency  exists  in  a  Reserve  Component  unit 


Functional  Area 
Importance 


The  amount  of  weight  that  the  deficiency  should  be  given  when 
evaluating  Aviation  Standardization  and  Training  in  a 
Reserve  Component  unit 


Mission  The  extent  to  which  a  Reserve  Component  unit  with  the  deficiency 

Criticality  can  perform  its  mobilization  mission 


Note:  When  making  your  ratings  of  the  characterictics,  assume 

that  the  deficiency  described  In  each  Item  Is  the  ONLY 
deficiency  that  exists  In  a  facility  or  In  a  unit. 


Please  turn  the  page  and  review  Sample  Items  1  and  2. 
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SAMPLE  ITEM  1 


Presented  below  is  a  checklist  Hem  that  has  been  used  to  evaluate  Ground  Tratdc  Control  in  an  aviation 
support  facility  or  in  a  Reserve  Component  (RC)  aviation  unit .  Check  [  V  ]  the  response  alternative  in  rating 
Hems  1  •  6  that  you  consider  to  be  most  appropriate. 


Padastrlan  walkways  to  Operations  Area 
ware  not  marked  clearly. 


Note:  Rating  Items  1-3  Apply  to  a  Facility 


t.  How  much  effort  would  it  take  to  detect  the  deficiency  described  in  this  Hem  when  evaluating  a  facility 
during  an  ARMS  visit? 

(M  (2]  13)  14)  (5) 

It  would  take  almost  It  would  take  a  moderate,  but  It  would  take  a  great 

no  effort  to  detect  rtot  an  extensive  amount  of  deal  of  effort  to  detect 

this  deficiency  effort  to  detect  this  deficiency  this  deficiency 


2.  How  much  weight  should  the  deficiency  described  in  this  Hem  be  given  when  evaluating  Ground  Traffic  Control  in  a 

facility^  . 

(1)  [21  (3)  [41  (g/f 

The  defidetxry  should  The  deficiency  should  The  deficiency  should 

be  given  little  be  given  a  moderate  be  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 

3.  To  what  extent  could  a  facility  with  the  deficiency  described  in  this  Hem  support  the  training  of  a  Reserve 
Component  unH? 

in  I2I  (3)  IV/f  IS} 

The  facility  could  The  facility  could  The  facility  could 

support  very  few  aspects  support  40-60%  supp>ort  nearly  alt  aspects 

of  unit  training  of  unit  training  of  unit  training 


Note:  Rating  Items  4  •  6  Apply  to  a  Unit 


4.  How  much  effort  would  it  take  to  detect  the  deficiency  described  in  this  item  when  evaluating  a  unit 
during  an  ARMS  visit? 

IM  12)  [3)  (4)  (51 

It  would  take  almost  it  would  take  a  moderate,  but  ft  would  take  a  great  deal 

no  effort  to  detect  not  an  extensive  amount  of  of  effort  to  detect 

this  deficiency  effort  to  detect  this  deficiency  this  deficiency 


5.  How  much  weight  <!^oiild  the  deficiency  described  in  this  Hem  be  given  when  evaluating  Ground  Traffic  Control  in  a 
unit? 

Ml  I  2  1  m  [  4  1  I  5  1 

The  deficiency  should  The  defidercy  should  be  The  deficiency  should  be 

be  given  little  given  a  moderate  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


(51 

The  deficiency  should  be 
given  a  great  deal 
of  weight 


6.  To  what  extant  could  a  unit  with  the  deficiency  described  in  this  Hem  perform  Hs  mobilization  mission  in  a 
satisfactory  manner  ?  ^ 

Ml  (21  [31  [41  [H 

The  unit  could  perfomt  The  unit  could  perform  The  unit  could  perform 

very  few  of  its  40-60%  of  its  almost  all  of  Hs 

mobilization  tasks  mobilization  tasks  mobilization  tasks 
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SAMPLE  ITEM  2 


Presented  below  is  a  checklist  item  that  has  been  used  to  evaluate  CommunlCMtlons  Management  in  an 
aviation  support  facility  or  in  a  Reserve  Component  (RC)  aviation  unit .  Check  [  V  ]  the  response  alternative  in 
rating  Kerns  1*6  that  you  consider  to  be  most  appropriate. 


Field  raeUoa  ware  not  kept  In  etete  of  repair. 


Note:  Rating  items  1  •  3  Apply  to  a  Facility 


1.  How  much  effort  would  K  take  to  detect  the  defidertcy  described  in  this  hem  when  evaluating  a  facility 


(11 

It  would  take  almost 
no  effort  to  detect 
this  defidency 


19^ 

It  would  take  a  moderate,  but 
rtot  an  extensive  amount  of 
effort  to  detect  this  defidency 


[5] 

It  would  take  a  great 
deal  of  effort  to  detect 
this  defidency 


Z  How  much  weight  should  the  deficiency  described  in  this  Kam  be  given  when  evaluating  Communications  Management 
in  a  facility?  ^ 

[1]  W  131  [4]  [51 

The  defiderxry  should  The  defidency  should  The  deficiency  should 

be  given  little  be  given  a  moderale  be  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


3.  To  what  extent  could  a  facility  with  the  defidency  described  in  this  item  support  the  training  of  a  Reserve 
Component  unit? 

[11  [21  [H  [41  [51 

The  fadlity  could  The  fadlity  could  The  fadlity  could 

support  very  few  aspects  support  40-60%  support  nearly  all  aspeds 

of  unit  training  of  unit  training  of  unit  training 


Note;  Rating  Items  4-6  Apply  to  a  Unit 


4.  How  much  effort  would  it  take  to  detect  the  defidency  described  in  this  Kem  when  evaluating  a  unit 


jrino  an  ARMS  visit? 

[1  J 

It  would  take  almost 
no  effort  to  detect 
this  defidency 


[31 

It  would  take  a  moderate,  but 
not  an  extensive  amount  of 
effort  to  detect  this  deficiency 


[5] 

it  would  take  a  great  deal 
of  effort  to  detect 
this  defidency 


5.  How  much  weight  should  the  deficiency  descrbed  in  this  Kem  be  given  when  evaluating  Communications  Management 
in  a  unit? 

[11  [21  [31  [41  [Vf 

The  deficiency  should  The  defidency  should  be  The  defidency  should  be 

be  given  liKle  given  a  moderate  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


The  defidency  should  be 
given  a  great  deal 
of  weight 


6.  To  what  extent  could  a  unit  with  the  defidency  described  in  this  item  perform  Ks  mobilization  mission  in  a 
satisfactory  manner  ?  y 

[11  [M  (31  [41  [51 

The  unit  could  perform  The  unK  could  perform  The  unit  could  perform 

very  few  of  its  40-60%  of  its  almost  all  of  its 

mobilization  tasks  mobilization  tasks  mobilization  tasks 
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EXPLANATION  OF  SAMPLE  ITEMS 


Sample  Items  1  and  2  show  you  how  an  SME  might  use  the  rating  scales.  Both  Sample 
Items  1  and  2  are  fictitious,  and  are  not  part  of  tiie  First  Army  ARMS  Cl  ^cktist  Likewise, 
Communications  Management  and  Ground  Traffic  Control  are  not  functional  areas  evaluated 
during  an  ARMS.  The  sample  items  are  provided  to  demonstrate  two  important  points  about  the 
the  rating  task. 

F/rst,  the  deficiency  described  In  the  Item  may  be  Important  for 
evaluating  a  facility's  or  unit's  status  In  a  functional  area,  but 
may  not  be  critical  to  the  capability  of  the  facility  or  unit  to 
accomplish  Its  mission. 

In  Sample  Item  1 ,  the  SME  judged  that  having  pedestrian  walkways  to  the  Operations  Area  that 
were  not  marked  clearly  was  important  for  the  evaluation  of  Ground  Traffic  Control  in  the 
facility  (see  rating  item  2),  but  that  the  deficiency  was  not  critical  to  the  facility's 
capability  of  supporting  the  training  of  a  Reserve  Component  unit  (see  rating  item  3). 


Second,  the  Item  Detectability,  Functional  Area  Importance,  and 
Mission  Criticality  of  the  deficiency  described  In  an  Item  may  be 
different  for  a  facility  than  for  a  unit. 


In  Sample  Item  2,  the  SME  judged  that  not  having  field  radios  in  a  state  of  repair  was  more 
important  for  evaluating  Communications  Management  in  a  unit  (see  rating  item  5)  than  for 
evaluating  Communications  Management  in  a  facility  (see  rating  item  2). 


Once  you  have  finished  reviewing  the  sample  items, 
please  turn  to  page  1 1  and  begin  the  rating  task. 


Presented  below  is  a  checklist  item  that  has  been  used  to  evaluate  AIrcraft/Fllght  Lln»  Operations 
in  an  aviation  support  facility  or  in  a  Reserve  Component  (RC)  aviation  unit .  Check  [  V  ]  the  response 
alternative  in  rating  items  1*6  that  you  consider  to  be  most  appropriate. 


Appropriate  numbers  of  first  eld  kits  were 
not  available  In  each  aircraft. 


Note:  Rating  Items  1-3  Apply  to  a  Facility 


1.  How  much  effort  would  it  take  to  detect  the  deficiency  described  in  this  item  when  evaluating  a 
facility  during  an  ARMS  visit? 

til  (21  C3]  14]  [5] 

It  would  take  almost  it  would  take  a  moderate,  but  It  would  take  a  great 

no  effort  to  detect  rtot  an  extensive  amount  of  deal  of  effort  to  detect 

this  defidency  effort  to  detect  this  deficiency  this  deficiency 


[5] 

It  would  take  a  great 
deal  of  effort  to  detect 
this  deficiency 


2.  How  much  weight  should  the  deficiency  described  in  this  item  be  given  when  evaluating  Aircraft/Flight 
Line  Operations  in  a  facility? 

Ml  121  [3]  (4)  [51 

The  deficiency  should  The  deficiency  should  The  deficiency  should 

be  given  little  be  given  a  m^erate  be  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


3.  To  what  extent  could  a  facility  with  the  deficiency  described  in  this  item  support  the  training  of  a 
Reserve  Component  unit? 


Reserve  Component  unit? 

Ml 

The  fadlity  could 
support  very  few  aspects 
of  unit  training 


131 

The  fadlity  could 
support  40-60% 
of  unit  training 


[5] 

The  fadlity  could 
support  nearly  all  aspects 
of  unit  training 


Note;  Rating  Items  4  •  6  Apply  to  a  Unit 


4.  How  much  effort  would  it  take  to  detect  the  deficiency  described  in  this  item  when  evaluating  a  unit 


Jurino  an  ARMS  visit? 
[11 

It  would  take  almost 
no  effort  to  detect 
this  defidency 


[31 

It  would  take  a  moderate,  but 
not  an  extensive  amount  of 
effort  to  detect  this  deficiency 


[5] 

H  would  take  a  great  deal 
of  effort  to  detect 
this  defidency 


5.  How  much  weight  should  the  deficiency  described  in  this  item  be  given  when  evaluating  Aircraft/Flight 
Line  Operations  in  a  unit? 

Ml  [2]  [3]  [4]  [5] 

The  defidency  should  The  defidency  should  be  The  deficiency  should  be 

be  given  little  given  a  moderate  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


[51 

The  deficiency  should  be 
given  a  great  deal 
of  weight 


6.  To  what  extent  could  a  unit  with  the  deficiency  described  in  this  item  perform  its  mobilization  mission  in 
a  satisfactory  manner  ? 

Ml  [21  [3]  [4]  [5] 

The  unit  could  perform  The  unit  could  perform  The  unit  could  perform 

very  few  of  its  40-60%  of  its  almost  all  of  its 

mobilization  tasks  mobilization  tasks  mobilization  tasks 


B-8 


Presented  below  is  a  checklist  Hern  that  has  been  used  to  evaluate  AIrcratt/Fllght  UnB  Operation 
in  an  aviation  support  facility  or  in  a  Reserve  Component  (RC)  aviation  unit .  Check  [  V  ]  the  response 
alternative  in  rating  items  1-6  that  you  consider  to  be  most  appropriate. 


Appropriate  numbers  of  tire  extinguishers 
were  not  available  In  each  aircraft. 


Note:  Rating  Items  1-3  Apply  to  a  Facility 


How  much  effort  would  it  take  to 
facility  during  an  ARMS  visit? 
11)  [21 
It  would  take  almost 
no  effort  to  detect 
this  deficiency 


detect  the  deficiency  described  in  this  item  when  evaluating  a 


[3] 

It  would  take  a  moderate,  but 
not  an  extensive  amount  of 
effort  to  detect  diis  deficiency 


[4]  (51 

h  would  take  a  great 
deal  of  effort  to  detect 
this  deficiency 


2.  How  much  vreight  should  the  deficiency  described  in  this  item  be  given  when  evaluating  Avcraft/Flight 
Line  Operation  in  a  facility^ 

in  [2]  [31  [41  [51 

The  deficiency  should  The  deficiency  should  The  deficiency  should 

be  given  little  be  given  a  moderate  be  given  a  great  deal 

or  no  weight  amount  of  weight  of  weight 


3. 


To  what  extent  could  a  facility  with  the  defidertcy  described  in  this  Hern  support  the  training  of  a 
Reserve  Component  unit? 

(21  [31  [41 

The  fadlity  could 
support  40-60% 
of  unit  training 


in 

The  fadlity  could 
support  very  few  aspects 
of  unit  training 


[51 

The  fadlity  could 
support  nearly  all  aspects 
of  unit  training 


Note;  Rating  Items  4-6  Apply  to  a  Unit 


4. 


How  much  effort  would  it  take  to  detect  the  defidency  described  in  this 
during  an  ARMS  visit? 

in  [2]  [31  [4 

It  would  take  almost  It  would  take  a  moderate,  but 

no  effort  to  detect  not  an  extensive  amount  of 

this  defidency  effort  to  detect  this  defidency 


Hem  when  evaluating  a  unit 

]  [51 

n  would  take  a  great  deal 
of  effort  to  detect 
this  defidency 


5. 


How  much  weight  should  the  deficiency  described  in  this  item  be  given 

Line  Operation  in  a  unit? 

[1]  [21 

[3J 

[4 

The  defidency  should 

The  defidency  should  be 

be  given  little 

given  a  moderate 

or  no  weight 

amount  of  weight 

when  evaluating  Aircraft/Flight 

)  [51 

The  deficiency  should  be 
given  a  great  deal 
of  weight 


6. 


To  what  extent  could  a  unit 
a  salisfadory  manner  ? 

[1  1 

The  unit  could  perform 
very  few  of  its 
mobilization  tasks 


with  the  deficiency  described  in  this  Hem  perform  its  mobilization  mission  in 


[21  [31  [41 

The  unH  could  perform 
40-60%  of  its 
mobilization  tasks 


[51 

The  unH  could  perform 
almost  all  of  Hs 
mobilization  tasks 


WL0002 


B-9 


You  have  completed  all  the  Items  In  the  questionnaire  booklet. 


Again,  thank  you  for  your  cooperation. 


Please  mall  the  questionnaire  booklet  to: 

ARI  Aviation  Research  and  Development  Activity 
ATTN:  PERNR  (ARMS) 

Fort  Rucker,  Alabama  36362-5354 
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